<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Sphinx 1.10-beta reference manual</title><meta name="generator" content="DocBook XSL Stylesheets V1.70.1">
<style type="text/css">pre.programlisting { background-color: #f0f0f0; padding: 0.5em; margin-left: 2em; margin-right: 2em; }</style>
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="book" lang="en"><div class="titlepage"><div><div><h1 class="title">Sphinx 1.10-beta reference manual</h1></div>
<div><h2 class="subtitle">Free open-source SQL full-text search engine</h2></div>
<div><p class="copyright">Copyright &copy; 2001-2010 Andrew Aksyonoff</p></div>
<div><p class="copyright">Copyright &copy; 2008-2010 Sphinx Technologies Inc, <a href="http://sphinxsearch.com" target="_top">http://sphinxsearch.com</a></p></div></div>
<hr></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="chapter"><a href="#intro">1. Introduction</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#about">1.1. About</a></span></dt>
<dt><span class="sect1"><a href="#features">1.2. Sphinx features</a></span></dt>
<dt><span class="sect1"><a href="#getting">1.3. Where to get Sphinx</a></span></dt>
<dt><span class="sect1"><a href="#license">1.4. License</a></span></dt>
<dt><span class="sect1"><a href="#credits">1.5. Credits</a></span></dt>
<dt><span class="sect1"><a href="#history">1.6. History</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#installation">2. Installation</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#supported-system">2.1. Supported systems</a></span></dt>
<dt><span class="sect1"><a href="#required-tools">2.2. Required tools</a></span></dt>
<dt><span class="sect1"><a href="#installing">2.3. Installing Sphinx on Linux</a></span></dt>
<dt><span class="sect1"><a href="#installing-windows">2.4. Installing Sphinx on Windows</a></span></dt>
<dt><span class="sect1"><a href="#install-problems">2.5. Known installation issues</a></span></dt>
<dt><span class="sect1"><a href="#quick-tour">2.6. Quick Sphinx usage tour</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#indexing">3. Indexing</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#sources">3.1. Data sources</a></span></dt>
<dt><span class="sect1"><a href="#attributes">3.2. Attributes</a></span></dt>
<dt><span class="sect1"><a href="#mva">3.3. MVA (multi-valued attributes)</a></span></dt>
<dt><span class="sect1"><a href="#indexes">3.4. Indexes</a></span></dt>
<dt><span class="sect1"><a href="#data-restrictions">3.5. Restrictions on the source data</a></span></dt>
<dt><span class="sect1"><a href="#charsets">3.6. Charsets, case folding, and translation tables</a></span></dt>
<dt><span class="sect1"><a href="#sql">3.7. SQL data sources (MySQL, PostgreSQL)</a></span></dt>
<dt><span class="sect1"><a href="#xmlpipe">3.8. xmlpipe data source</a></span></dt>
<dt><span class="sect1"><a href="#xmlpipe2">3.9. xmlpipe2 data source</a></span></dt>
<dt><span class="sect1"><a href="#live-updates">3.10. Live index updates</a></span></dt>
<dt><span class="sect1"><a href="#delta-updates">3.11. Delta index updates</a></span></dt>
<dt><span class="sect1"><a href="#index-merging">3.12. Index merging</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#rt-indexes">4. Real-time indexes</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#rt-overview">4.1. RT indexes overview</a></span></dt>
<dt><span class="sect1"><a href="#rt-caveats">4.2. Known caveats with RT indexes</a></span></dt>
<dt><span class="sect1"><a href="#rt-internals">4.3. RT index internals</a></span></dt>
<dt><span class="sect1"><a href="#rt-binlog">4.4. Binary logging</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#searching">5. Searching</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#matching-modes">5.1. Matching modes</a></span></dt>
<dt><span class="sect1"><a href="#boolean-syntax">5.2. Boolean query syntax</a></span></dt>
<dt><span class="sect1"><a href="#extended-syntax">5.3. Extended query syntax</a></span></dt>
<dt><span class="sect1"><a href="#weighting">5.4. Weighting</a></span></dt>
<dt><span class="sect1"><a href="#sorting-modes">5.5. Sorting modes</a></span></dt>
<dt><span class="sect1"><a href="#clustering">5.6. Grouping (clustering) search results </a></span></dt>
<dt><span class="sect1"><a href="#distributed">5.7. Distributed searching</a></span></dt>
<dt><span class="sect1"><a href="#query-log-format">5.8. <code class="filename">searchd</code> query log format</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql">5.9. MySQL protocol support and SphinxQL</a></span></dt>
<dt><span class="sect1"><a href="#multi-queries">5.10. Multi-queries</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#command-line-tools">6. Command line tools reference</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#ref-indexer">6.1. <code class="filename">indexer</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-searchd">6.2. <code class="filename">searchd</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-search">6.3. <code class="filename">search</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-spelldump">6.4. <code class="filename">spelldump</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-indextool">6.5. <code class="filename">indextool</code> command reference</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#sphinxql-reference">7. SphinxQL reference</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#sphinxql-select">7.1. SELECT syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-meta">7.2. SHOW META syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-warnings">7.3. SHOW WARNINGS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-status">7.4. SHOW STATUS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-insert">7.5. INSERT and REPLACE syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-delete">7.6. DELETE syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-set">7.7. SET syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-commit">7.8. BEGIN, COMMIT, and ROLLBACK syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-call-snippets">7.9. CALL SNIPPETS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-call-keywords">7.10. CALL KEYWORDS syntax</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#api-reference">8. API reference</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#api-funcgroup-general">8.1. General API functions</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-getlasterror">8.1.1. GetLastError</a></span></dt>
<dt><span class="sect2"><a href="#api-func-getlastwarning">8.1.2. GetLastWarning</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setserver">8.1.3. SetServer</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setretries">8.1.4. SetRetries</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setconnecttimeout">8.1.5. SetConnectTimeout</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setarrayresult">8.1.6. SetArrayResult</a></span></dt>
<dt><span class="sect2"><a href="#api-func-isconnecterror">8.1.7. IsConnectError</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-general-query-settings">8.2. General query settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setlimits">8.2.1. SetLimits</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setmaxquerytime">8.2.2. SetMaxQueryTime</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setoverride">8.2.3. SetOverride</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setselect">8.2.4. SetSelect</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-fulltext-query-settings">8.3. Full-text search query settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setmatchmode">8.3.1. SetMatchMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setrankingmode">8.3.2. SetRankingMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setsortmode">8.3.3. SetSortMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setweights">8.3.4. SetWeights</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfieldweights">8.3.5. SetFieldWeights</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setindexweights">8.3.6. SetIndexWeights</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-filtering">8.4. Result set filtering settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setidrange">8.4.1. SetIDRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilter">8.4.2. SetFilter</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilterrange">8.4.3. SetFilterRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilterfloatrange">8.4.4. SetFilterFloatRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setgeoanchor">8.4.5. SetGeoAnchor</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-groupby">8.5. GROUP BY settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setgroupby">8.5.1. SetGroupBy</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setgroupdistinct">8.5.2. SetGroupDistinct</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-querying">8.6. Querying</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-query">8.6.1. Query</a></span></dt>
<dt><span class="sect2"><a href="#api-func-addquery">8.6.2. AddQuery</a></span></dt>
<dt><span class="sect2"><a href="#api-func-runqueries">8.6.3. RunQueries</a></span></dt>
<dt><span class="sect2"><a href="#api-func-resetfilters">8.6.4. ResetFilters</a></span></dt>
<dt><span class="sect2"><a href="#api-func-resetgroupby">8.6.5. ResetGroupBy</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-additional-functionality">8.7. Additional functionality</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-buildexcerpts">8.7.1. BuildExcerpts</a></span></dt>
<dt><span class="sect2"><a href="#api-func-updateatttributes">8.7.2. UpdateAttributes</a></span></dt>
<dt><span class="sect2"><a href="#api-func-buildkeywords">8.7.3. BuildKeywords</a></span></dt>
<dt><span class="sect2"><a href="#api-func-escapestring">8.7.4. EscapeString</a></span></dt>
<dt><span class="sect2"><a href="#api-func-status">8.7.5. Status</a></span></dt>
<dt><span class="sect2"><a href="#api-func-flushattributes">8.7.6. FlushAttributes</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-pconn">8.8. Persistent connections</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-open">8.8.1. Open</a></span></dt>
<dt><span class="sect2"><a href="#api-func-close">8.8.2. Close</a></span></dt>
</dl></dd></dl></dd><dt><span class="chapter"><a href="#sphinxse">9. MySQL storage engine (SphinxSE)</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#sphinxse-overview">9.1. SphinxSE overview</a></span></dt>
<dt><span class="sect1"><a href="#sphinxse-installing">9.2. Installing SphinxSE</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#sphinxse-mysql50">9.2.1. Compiling MySQL 5.0.x with SphinxSE</a></span></dt>
<dt><span class="sect2"><a href="#sphinxse-mysql51">9.2.2. Compiling MySQL 5.1.x with SphinxSE</a></span></dt>
<dt><span class="sect2"><a href="#sphinxse-checking">9.2.3. Checking SphinxSE installation</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#sphinxse-using">9.3. Using SphinxSE</a></span></dt>
<dt><span class="sect1"><a href="#sphinxse-snippets">9.4. Building snippets (excerpts) via MySQL</a></span></dt>
</dl></dd><dt><span class="chapter"><a href="#reporting-bugs">10. Reporting bugs</a></span></dt>
<dt><span class="chapter"><a href="#conf-reference">11. <code class="filename">sphinx.conf</code> options reference</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#confgroup-source">11.1. Data source configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-source-type">11.1.1. type</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-host">11.1.2. sql_host</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-port">11.1.3. sql_port</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-user">11.1.4. sql_user</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-pass">11.1.5. sql_pass</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-db">11.1.6. sql_db</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-sock">11.1.7. sql_sock</a></span></dt>
<dt><span class="sect2"><a href="#conf-mysql-connect-flags">11.1.8. mysql_connect_flags</a></span></dt>
<dt><span class="sect2"><a href="#conf-mysql-ssl">11.1.9. mysql_ssl_cert, mysql_ssl_key, mysql_ssl_ca</a></span></dt>
<dt><span class="sect2"><a href="#conf-odbc-dsn">11.1.10. odbc_dsn</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-pre">11.1.11. sql_query_pre</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query">11.1.12. sql_query</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-joined-field">11.1.13. sql_joined_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-range">11.1.14. sql_query_range</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-range-step">11.1.15. sql_range_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-killlist">11.1.16. sql_query_killlist</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-uint">11.1.17. sql_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-bool">11.1.18. sql_attr_bool</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-bigint">11.1.19. sql_attr_bigint</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-timestamp">11.1.20. sql_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-str2ordinal">11.1.21. sql_attr_str2ordinal</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-float">11.1.22. sql_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-multi">11.1.23. sql_attr_multi</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-string">11.1.24. sql_attr_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-str2wordcount">11.1.25. sql_attr_str2wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-field-string">11.1.26. sql_field_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-field-str2wordcount">11.1.27. sql_field_str2wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-file-field">11.1.28. sql_file_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-post">11.1.29. sql_query_post</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-post-index">11.1.30. sql_query_post_index</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-ranged-throttle">11.1.31. sql_ranged_throttle</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-info">11.1.32. sql_query_info</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-command">11.1.33. xmlpipe_command</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field">11.1.34. xmlpipe_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field-string">11.1.35. xmlpipe_field_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field-wordcount">11.1.36. xmlpipe_field_wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-uint">11.1.37. xmlpipe_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-bool">11.1.38. xmlpipe_attr_bool</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-timestamp">11.1.39. xmlpipe_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-str2ordinal">11.1.40. xmlpipe_attr_str2ordinal</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-float">11.1.41. xmlpipe_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-multi">11.1.42. xmlpipe_attr_multi</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-string">11.1.43. xmlpipe_attr_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-fixup-utf8">11.1.44. xmlpipe_fixup_utf8</a></span></dt>
<dt><span class="sect2"><a href="#conf-mssql-winauth">11.1.45. mssql_winauth</a></span></dt>
<dt><span class="sect2"><a href="#conf-mssql-unicode">11.1.46. mssql_unicode</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-zlib">11.1.47. unpack_zlib</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-mysqlcompress">11.1.48. unpack_mysqlcompress</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-mysqlcompress-maxsize">11.1.49. unpack_mysqlcompress_maxsize</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-index">11.2. Index configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-index-type">11.2.1. type</a></span></dt>
<dt><span class="sect2"><a href="#conf-source">11.2.2. source</a></span></dt>
<dt><span class="sect2"><a href="#conf-path">11.2.3. path</a></span></dt>
<dt><span class="sect2"><a href="#conf-docinfo">11.2.4. docinfo</a></span></dt>
<dt><span class="sect2"><a href="#conf-mlock">11.2.5. mlock</a></span></dt>
<dt><span class="sect2"><a href="#conf-morphology">11.2.6. morphology</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-stemming-len">11.2.7. min_stemming_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-stopwords">11.2.8. stopwords</a></span></dt>
<dt><span class="sect2"><a href="#conf-wordforms">11.2.9. wordforms</a></span></dt>
<dt><span class="sect2"><a href="#conf-exceptions">11.2.10. exceptions</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-word-len">11.2.11. min_word_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-charset-type">11.2.12. charset_type</a></span></dt>
<dt><span class="sect2"><a href="#conf-charset-table">11.2.13. charset_table</a></span></dt>
<dt><span class="sect2"><a href="#conf-ignore-chars">11.2.14. ignore_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-prefix-len">11.2.15. min_prefix_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-infix-len">11.2.16. min_infix_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-prefix-fields">11.2.17. prefix_fields</a></span></dt>
<dt><span class="sect2"><a href="#conf-infix-fields">11.2.18. infix_fields</a></span></dt>
<dt><span class="sect2"><a href="#conf-enable-star">11.2.19. enable_star</a></span></dt>
<dt><span class="sect2"><a href="#conf-ngram-len">11.2.20. ngram_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-ngram-chars">11.2.21. ngram_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-phrase-boundary">11.2.22. phrase_boundary</a></span></dt>
<dt><span class="sect2"><a href="#conf-phrase-boundary-step">11.2.23. phrase_boundary_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-strip">11.2.24. html_strip</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-index-attrs">11.2.25. html_index_attrs</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-remove-elements">11.2.26. html_remove_elements</a></span></dt>
<dt><span class="sect2"><a href="#conf-local">11.2.27. local</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent">11.2.28. agent</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-blackhole">11.2.29. agent_blackhole</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-connect-timeout">11.2.30. agent_connect_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-query-timeout">11.2.31. agent_query_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-preopen">11.2.32. preopen</a></span></dt>
<dt><span class="sect2"><a href="#conf-ondisk-dict">11.2.33. ondisk_dict</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-enable">11.2.34. inplace_enable</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-hit-gap">11.2.35. inplace_hit_gap</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-docinfo-gap">11.2.36. inplace_docinfo_gap</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-reloc-factor">11.2.37. inplace_reloc_factor</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-write-factor">11.2.38. inplace_write_factor</a></span></dt>
<dt><span class="sect2"><a href="#conf-index-exact-words">11.2.39. index_exact_words</a></span></dt>
<dt><span class="sect2"><a href="#conf-overshort-step">11.2.40. overshort_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-stopword-step">11.2.41. stopword_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-hitless-words">11.2.42. hitless_words</a></span></dt>
<dt><span class="sect2"><a href="#conf-expand-keywords">11.2.43. expand_keywords</a></span></dt>
<dt><span class="sect2"><a href="#conf-blend-chars">11.2.44. blend_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-mem-limit">11.2.45. rt_mem_limit</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-field">11.2.46. rt_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-uint">11.2.47. rt_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-bigint">11.2.48. rt_attr_bigint</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-float">11.2.49. rt_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-timestamp">11.2.50. rt_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-string">11.2.51. rt_attr_string</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-indexer">11.3. <code class="filename">indexer</code> program configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-mem-limit">11.3.1. mem_limit</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-iops">11.3.2. max_iops</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-iosize">11.3.3. max_iosize</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-xmlpipe2-field">11.3.4. max_xmlpipe2_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-write-buffer">11.3.5. write_buffer</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-file-field-buffer">11.3.6. max_file_field_buffer</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-searchd">11.4. <code class="filename">searchd</code> program configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-listen">11.4.1. listen</a></span></dt>
<dt><span class="sect2"><a href="#conf-address">11.4.2. address</a></span></dt>
<dt><span class="sect2"><a href="#conf-port">11.4.3. port</a></span></dt>
<dt><span class="sect2"><a href="#conf-log">11.4.4. log</a></span></dt>
<dt><span class="sect2"><a href="#conf-query-log">11.4.5. query_log</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-timeout">11.4.6. read_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-client-timeout">11.4.7. client_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-children">11.4.8. max_children</a></span></dt>
<dt><span class="sect2"><a href="#conf-pid-file">11.4.9. pid_file</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-matches">11.4.10. max_matches</a></span></dt>
<dt><span class="sect2"><a href="#conf-seamless-rotate">11.4.11. seamless_rotate</a></span></dt>
<dt><span class="sect2"><a href="#conf-preopen-indexes">11.4.12. preopen_indexes</a></span></dt>
<dt><span class="sect2"><a href="#conf-unlink-old">11.4.13. unlink_old</a></span></dt>
<dt><span class="sect2"><a href="#conf-attr-flush-period">11.4.14. attr_flush_period</a></span></dt>
<dt><span class="sect2"><a href="#conf-ondisk-dict-default">11.4.15. ondisk_dict_default</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-packet-size">11.4.16. max_packet_size</a></span></dt>
<dt><span class="sect2"><a href="#conf-mva-updates-pool">11.4.17. mva_updates_pool</a></span></dt>
<dt><span class="sect2"><a href="#conf-crash-log-path">11.4.18. crash_log_path</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-filters">11.4.19. max_filters</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-filter-values">11.4.20. max_filter_values</a></span></dt>
<dt><span class="sect2"><a href="#conf-listen-backlog">11.4.21. listen_backlog</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-buffer">11.4.22. read_buffer</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-unhinted">11.4.23. read_unhinted</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-batch-queries">11.4.24. max_batch_queries</a></span></dt>
<dt><span class="sect2"><a href="#conf-subtree-docs-cache">11.4.25. subtree_docs_cache</a></span></dt>
<dt><span class="sect2"><a href="#conf-subtree-hits-cache">11.4.26. subtree_hits_cache</a></span></dt>
<dt><span class="sect2"><a href="#conf-workers">11.4.27. workers</a></span></dt>
<dt><span class="sect2"><a href="#conf-dist-threads">11.4.28. dist_threads</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-path">11.4.29. binlog_path</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-flush">11.4.30. binlog_flush</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-max-log-size">11.4.31. binlog_max_log_size</a></span></dt>
</dl></dd></dl></dd><dt><span class="appendix"><a href="#changelog">A. Sphinx revision history</a></span></dt>
<dd><dl><dt><span class="sect1"><a href="#rel110">A.1. Version 1.10-beta, 19 jul 2010</a></span></dt>
<dt><span class="sect1"><a href="#rel099">A.2. Version 0.9.9-release, 02 dec 2009</a></span></dt>
<dt><span class="sect1"><a href="#rel099rc2">A.3. Version 0.9.9-rc2, 08 apr 2009</a></span></dt>
<dt><span class="sect1"><a href="#rel099rc1">A.4. Version 0.9.9-rc1, 17 nov 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel0981">A.5. Version 0.9.8.1, 30 oct 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel098">A.6. Version 0.9.8, 14 jul 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel097">A.7. Version 0.9.7, 02 apr 2007</a></span></dt>
<dt><span class="sect1"><a href="#rel097rc2">A.8. Version 0.9.7-rc2, 15 dec 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel097rc">A.9. Version 0.9.7-rc1, 26 oct 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel096">A.10. Version 0.9.6, 24 jul 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel096rc1">A.11. Version 0.9.6-rc1, 26 jun 2006</a></span></dt>
</dl></dd></dl></div>
<div class="list-of-examples"><p><b>List of Examples</b></p><dl><dt>3.1. <a href="#ex-ranged-queries">Ranged query usage example</a></dt>
<dt>3.2. <a href="#ex-xmlpipe-document">XMLpipe document stream</a></dt>
<dt>3.3. <a href="#ex-xmlpipe2-document">xmlpipe2 document stream</a></dt>
<dt>3.4. <a href="#ex-live-updates">Fully automated live updates</a></dt>
<dt>4.1. <a href="#ex-rt-updates">RT index declaration</a></dt>
<dt>5.1. <a href="#ex-boolean-query">Boolean query example</a></dt>
<dt>5.2. <a href="#ex-extended-query">Extended matching mode: query example</a></dt>
</dl></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="intro"></a>Chapter&nbsp;1.&nbsp;Introduction</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#about">1.1. About</a></span></dt>
<dt><span class="sect1"><a href="#features">1.2. Sphinx features</a></span></dt>
<dt><span class="sect1"><a href="#getting">1.3. Where to get Sphinx</a></span></dt>
<dt><span class="sect1"><a href="#license">1.4. License</a></span></dt>
<dt><span class="sect1"><a href="#credits">1.5. Credits</a></span></dt>
<dt><span class="sect1"><a href="#history">1.6. History</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="about"></a>1.1.&nbsp;About</h2></div></div></div>
<p>
Sphinx is a full-text search engine, publicly distributed under GPL version 2.
Commercial licensing (eg. for embedded use) is available upon request. 
</p><p>
Technically, Sphinx is a standalone software package provides
fast and relevant full-text search functionality to client applications.
It was specially designed to integrate well with SQL databases storing
the data, and to be easily accessed scripting languages. However, Sphinx
does not depend on nor require any specific database to function.
</p><p>
Applications can access Sphinx search daemon (searchd) using any of
the three different access methods: a) via native search API (SphinxAPI),
b) via Sphinx own implementation of MySQL network protocol (using a small
SQL subset called SphinxQL), or c) via MySQL server with a pluggable
storage engine (SphinxSE).
</p><p>
Official native SphinxAPI implementations for PHP, Perl, Ruby, and Java
are included within the distribution package. API is very lightweight
so porting it to a new language is known to take a few hours or days.
Third party API ports and plugins exist for Perl, C#, Haskell,
Ruby-on-Rails, and possibly other languages and frameworks.
</p><p>
Starting version 1.10-beta, Sphinx supports two different indexing
backends: "disk" index backend, and "realtime" (RT) index backend.
Disk indexes support online full-text index rebuilds, but online updates
can only be done on non-text (attribute) data.  RT indexes additionally
allow for online full-text index updates. Previous versions only
supported disk indexes.
</p><p>
Data can be loaded into disk indexes using a so-called data source.
Built-in sources can fetch data directly from MySQL, PostgreSQL, ODBC
compliant database (MS SQL, Oracle, etc), or a pipe in a custom XML format.
Adding new data sources drivers (eg. to natively support other DBMSes)
is designed to be as easy as possible. RT indexes, as of 1.10-beta,
can only be populated using SphinxQL.
</p><p>
As for the name, Sphinx is an acronym which is officially decoded
as SQL Phrase Index. Yes, I know about CMU's Sphinx project. 
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="features"></a>1.2.&nbsp;Sphinx features</h2></div></div></div>
<p>
Key Sphinx features are:
</p><div class="itemizedlist"><ul type="disc"><li>high indexing and searching performance;</li>
<li>advanced indexing and querying tools (flexible and feature-rich text tokenizer, querying language, several different ranking modes, etc);</li>
<li>advanced result set post-processing (SELECT with expressions, WHERE, ORDER BY, GROUP BY etc over text search results);</li>
<li>proven scalability up to billions of documents, terabytes of data, and thousands of queries per second;</li>
<li>easy integration with SQL and XML data sources, and SphinxAPI, SphinxQL, or SphinxSE search interfaces;</li>
<li>easy scaling with distributed searches.</li>
</ul></div>
<p>
To expand a bit, Sphinx:
</p><div class="itemizedlist"><ul type="disc"><li>has high indexing speed (upto 10-15 MB/sec per core on an internal benchmark);</li>
<li>has high search speed (upto 150-250 queries/sec per core against 1,000,000 documents, 1.2 GB of data on an internal benchmark);</li>
<li>has high scalability (biggest known cluster indexes over 3,000,000,000 documents, and busiest one peaks over 50,000,000 queries/day);</li>
<li>provides good relevance ranking through combination of phrase proximity ranking and statistical (BM25) ranking;</li>
<li>provides distributed searching capabilities;</li>
<li>provides document excerpts (snippets) generation;</li>
<li>provides searching from within application with SphinxAPI or SphinxQL interfaces, and from within MySQL with pluggable SphinxSE storage engine;</li>
<li>supports boolean, phrase, word proximity and other types of queries;</li>
<li>supports multiple full-text fields per document (upto 32 by default);</li>
<li>supports multiple additional attributes per document (ie. groups, timestamps, etc);</li>
<li>supports stopwords;</li>
<li>supports morphological word forms dictionaries;</li>
<li>supports tokenizing exceptions;</li>
<li>supports both single-byte encodings and UTF-8;</li>
<li>supports stemming (stemmers for English, Russian and Czech are built-in; and stemmers for
French, Spanish, Portuguese, Italian, Romanian, German, Dutch, Swedish, Norwegian, Danish, Finnish, Hungarian,
are available by building third party <a href="http://snowball.tartarus.org/" target="_top">libstemmer library</a>);</li>
<li>supports MySQL natively (all types of tables, including MyISAM, InnoDB, NDB, Archive, etc are supported);</li>
<li>supports PostgreSQL natively;</li>
<li>supports ODBC compliant databases (MS SQL, Oracle, etc) natively;</li>
<li>...has 50+ other features not listed here, refer to API and configuration manual!</li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="getting"></a>1.3.&nbsp;Where to get Sphinx</h2></div></div></div>
<p>Sphinx is available through its official Web site at <a href="http://sphinxsearch.com/" target="_top">http://sphinxsearch.com/</a>.
</p><p>Currently, Sphinx distribution tarball includes the following software:
</p><div class="itemizedlist"><ul type="disc"><li><code class="filename">indexer</code>: an utility which creates fulltext indexes;</li>
<li><code class="filename">search</code>: a simple command-line (CLI) test utility which searches through fulltext indexes;</li>
<li><code class="filename">searchd</code>: a daemon which enables external software (eg. Web applications) to search through fulltext indexes;</li>
<li><code class="filename">sphinxapi</code>: a set of searchd client API libraries for popular Web scripting languages (PHP, Python, Perl, Ruby).</li>
<li><code class="filename">spelldump</code>: a simple command-line tool to extract the items from an <code class="filename">ispell</code> or <code class="filename">MySpell</code>
(as bundled with OpenOffice) format dictionary to help customize your index, for use with <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a>.</li>
<li><code class="filename">indextool</code>: an utility to dump miscellaneous debug information about the index, added in version 0.9.9-rc2.</li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="license"></a>1.4.&nbsp;License</h2></div></div></div>
<p>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License,
or (at your option) any later version. See COPYING file for details.
</p><p>
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details. 
</p><p>
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 
</p><p>
Non-GPL licensing (for OEM/ISV embedded use) can also be arranged, please
<a href="http://sphinxsearch.com/contacts.html" target="_top">contact us</a> to discuss
commercial licensing possibilities.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="credits"></a>1.5.&nbsp;Credits</h2></div></div></div>
<h3>Author</h3><p>
Sphinx initial author (and a benevolent dictator ever since):
</p><div class="itemizedlist"><ul type="disc"><li>Andrew Aksyonoff, <a href="http://shodan.ru" target="_top">http://shodan.ru</a></li>
</ul></div>
<p>
</p><h3>Team</h3><p>
Past and present employees of Sphinx Technologies Inc who should be
noted on their work on Sphinx (in alphabetical order):
</p><div class="itemizedlist"><ul type="disc"><li>Alexander Klimenko</li>
<li>Alexey Dvoichenkov</li>
<li>Alexey Vinogradov</li>
<li>Ilya Kuznetsov</li>
<li>Stanislav Klinov</li>
</ul></div>
<p>
</p><h3>Contributors</h3><p>People who contributed to Sphinx and their contributions (in no particular order):
</p><div class="itemizedlist"><ul type="disc"><li>Robert "coredev" Bengtsson (Sweden), initial version of PostgreSQL data source</li>
<li>Len Kranendonk, Perl API</li>
<li>Dmytro Shteflyuk, Ruby API</li>
</ul></div>
<p>
</p><p>
Many other people have contributed ideas, bug reports, fixes, etc.
Thank you!
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="history"></a>1.6.&nbsp;History</h2></div></div></div>
<p>
Sphinx development was started back in 2001, because I didn't manage
to find an acceptable search solution (for a database driven Web site)
which would meet my requirements. Actually, each and every important aspect was a problem: 
</p><div class="itemizedlist"><ul type="disc"><li>search quality (ie. good relevance)
<div class="itemizedlist"><ul type="circle"><li>statistical ranking methods performed rather bad, especially on large collections of small documents (forums, blogs, etc)</li>
</ul></div></li>
<li>search speed
<div class="itemizedlist"><ul type="circle"><li>especially if searching for phrases which contain stopwords, as in "to be or not to be"</li>
</ul></div></li>
<li>moderate disk and CPU requirements when indexing
<div class="itemizedlist"><ul type="circle"><li>important in shared hosting enivronment, not to mention the indexing speed.</li>
</ul></div></li>
</ul></div>
<p>
</p><p>
Despite the amount of time passed and numerous improvements made in the
other solutions, there's still no solution which I personally would
be eager to migrate to. 
</p><p>
Considering that and a lot of positive feedback received from Sphinx users
during last years, the obvious decision is to continue developing Sphinx
(and, eventually, to take over the world).
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="installation"></a>Chapter&nbsp;2.&nbsp;Installation</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#supported-system">2.1. Supported systems</a></span></dt>
<dt><span class="sect1"><a href="#required-tools">2.2. Required tools</a></span></dt>
<dt><span class="sect1"><a href="#installing">2.3. Installing Sphinx on Linux</a></span></dt>
<dt><span class="sect1"><a href="#installing-windows">2.4. Installing Sphinx on Windows</a></span></dt>
<dt><span class="sect1"><a href="#install-problems">2.5. Known installation issues</a></span></dt>
<dt><span class="sect1"><a href="#quick-tour">2.6. Quick Sphinx usage tour</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="supported-system"></a>2.1.&nbsp;Supported systems</h2></div></div></div>
<p>
Most modern UNIX systems with a C++ compiler should be able
to compile and run Sphinx without any modifications.
</p><p>
Currently known systems Sphinx has been successfully running on are:
</p><div class="itemizedlist"><ul type="disc"><li>Linux 2.4.x, 2.6.x (many various distributions)</li>
<li>Windows 2000, XP</li>
<li>FreeBSD 4.x, 5.x, 6.x, 7.x</li>
<li>NetBSD 1.6, 3.0</li>
<li>Solaris 9, 11</li>
<li>Mac OS X</li>
</ul></div>
<p>
</p><p>
CPU architectures known to work include X86, X86-64, SPARC64, ARM.
</p><p>
Chance are good that Sphinx should work on other Unix platforms as well;
please report any platforms missing from this list that worked for you!
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="required-tools"></a>2.2.&nbsp;Required tools</h2></div></div></div>
<p>
On UNIX, you will need the following tools to build
and install Sphinx:
</p><div class="itemizedlist"><ul type="disc"><li>a working C++ compiler. GNU gcc is known to work.</li>
<li>a good make program. GNU make is known to work.</li>
</ul></div>
<p>
</p><p>
On Windows, you will need Microsoft Visual C/C++ Studio .NET 2003 or 2005.
Other compilers/environments will probably work as well, but for the
time being, you will have to build makefile (or other environment
specific project files) manually.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="installing"></a>2.3.&nbsp;Installing Sphinx on Linux</h2></div></div></div>
<div class="orderedlist"><ol type="1"><li><p>
	Extract everything from the distribution tarball (haven't you already?)
	and go to the <code class="filename">sphinx</code> subdirectory:
	</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;tar&nbsp;xzvf&nbsp;sphinx-0.9.8.tar.gz<br>
$&nbsp;cd&nbsp;sphinx<br>
</p></div>
</code></strong></p></li>
<li><p>Run the configuration program:</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;./configure</p></div>
</code></strong></p><p>
	There's a number of options to configure. The complete listing may
	be obtained by using <code class="option">--help</code> switch. The most important ones are:
	</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--prefix</code>, which specifies where to install Sphinx; such as <code class="option">--prefix=/usr/local/sphinx</code> (all of the examples use this prefix)</li>
<li><code class="option">--with-mysql</code>, which specifies where to look for MySQL
			include and library files, if auto-detection fails;</li>
<li><code class="option">--with-pgsql</code>, which specifies where to look for PostgreSQL
			include and library files.</li>
</ul></div>
<p>
	</p></li>
<li><p>Build the binaries:</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;make</p></div>
</code></strong></p></li>
<li><p>Install the binaries in the directory of your choice:
	(defaults to <code class="filename">/usr/local/bin/</code> on *nix systems,
	but is overridden with <code class="option">configure --prefix</code>)</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;make&nbsp;install</p></div>
</code></strong></p></li>
</ol></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="installing-windows"></a>2.4.&nbsp;Installing Sphinx on Windows</h2></div></div></div>
<p>Installing Sphinx on a Windows server is often easier than installing on a Linux environment;
unless you are preparing code patches, you can use the pre-compiled binary files from the Downloads
area on the website.</p><div class="orderedlist"><ol type="1"><li><p>Extract everything from the .zip file you have downloaded - <code class="filename">sphinx-0.9.8-win32.zip</code>
	(or <code class="filename">sphinx-0.9.8-win32-pgsql.zip</code> if you need PostgresSQL support as well.) You can use
	Windows Explorer in Windows XP and up to extract the files, or a freeware package like 7Zip to open the archive.</p><p>For the remainder of this guide, we will assume that the folders are unzipped into <code class="filename">C:\Sphinx</code>,
	such that <code class="filename">searchd.exe</code> can be found in <code class="filename">C:\Sphinx\bin\searchd.exe</code>. If you decide
	to use any different location for the folders or configuration file, please change it accordingly.</p></li>
<li><p>Edit the contents of sphinx.conf.in - specifically entries relating to @CONFDIR@ - to paths suitable for your system.</p></li>
<li><p>Install the <code class="filename">searchd</code> system as a Windows service:</p><p><strong class="userinput"><code>C:\Sphinx\bin&gt; C:\Sphinx\bin\searchd --install --config C:\Sphinx\sphinx.conf.in --servicename SphinxSearch</code></strong></p></li>
<li><p>The <code class="filename">searchd</code> service will now be listed in the Services panel
	within the Management Console, available from Administrative Tools. It will not have been
	started, as you will need to configure it and build your indexes with <code class="filename">indexer</code>
	before starting the service. A guide to do this can be found under
	<a href="#quick-tour" title="2.6.&nbsp;Quick Sphinx usage tour">Quick tour</a>.</p><p>During the next steps of the install (which involve running indexer pretty much as
	you would on Linux) you may find that you get an error relating to libmysql.dll not being found.
	If you have MySQL installed, you should find a copy of this library in your Windows directory,
	or sometimes in Windows\System32, or failing that in the MySQL core directories. If you
	do receive an error please copy libmysql.dll into the bin directory.</p></li>
</ol></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="install-problems"></a>2.5.&nbsp;Known installation issues</h2></div></div></div>
<p>
If <code class="filename">configure</code> fails to locate MySQL headers and/or libraries,
try checking for and installing <code class="filename">mysql-devel</code> package. On some systems,
it is not installed by default.
</p><p>
If <code class="filename">make</code> fails with a message which look like
</p><pre class="programlisting">
/bin/sh: g++: command not found
make[1]: *** [libsphinx_a-sphinx.o] Error 127
</pre><p>
try checking for and installing <code class="filename">gcc-c++</code> package.
</p><p>
If you are getting compile-time errors which look like
</p><pre class="programlisting">
sphinx.cpp:67: error: invalid application of `sizeof' to
    incomplete type `Private::SizeError&lt;false&gt;'
</pre><p>
this means that some compile-time type size check failed.
The most probable reason is that off_t type is less than 64-bit
on your system. As a quick hack, you can edit sphinx.h and replace off_t
with DWORD in a typedef for SphOffset_t, but note that this will prohibit
you from using full-text indexes larger than 2 GB. Even if the hack helps,
please report such issues, providing the exact error message and
compiler/OS details, so I could properly fix them in next releases.
</p><p>
If you keep getting any other error, or the suggestions above
do not seem to help you, please don't hesitate to contact me.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="quick-tour"></a>2.6.&nbsp;Quick Sphinx usage tour</h2></div></div></div>
<p>
All the example commands below assume that you installed Sphinx
in <code class="filename">/usr/local/sphinx</code>, so <code class="filename">searchd</code> can
be found in <code class="filename">/usr/local/sphinx/bin/searchd</code>.
</p><p>
To use Sphinx, you will need to:
</p><div class="orderedlist"><ol type="1"><li><p>Create a configuration file.</p><p>
	Default configuration file name is <code class="filename">sphinx.conf</code>.
	All Sphinx programs look for this file in current working directory
	by default.
	</p><p>
	Sample configuration file, <code class="filename">sphinx.conf.dist</code>, which has
	all the options documented, is created by <code class="filename">configure</code>.
	Copy and edit that sample file to make your own configuration: (assuming Sphinx is installed into <code class="filename">/usr/local/sphinx/</code>)
	</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;cd&nbsp;/usr/local/sphinx/etc<br>
$&nbsp;cp&nbsp;sphinx.conf.dist&nbsp;sphinx.conf<br>
$&nbsp;vi&nbsp;sphinx.conf</p></div>
</code></strong></p><p>
	Sample configuration file is setup to index <code class="filename">documents</code>
	table from MySQL database <code class="filename">test</code>; so there's <code class="filename">example.sql</code>
	sample data file to populate that table with a few documents for testing purposes:
	</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;mysql&nbsp;-u&nbsp;test&nbsp;&lt;&nbsp;/usr/local/sphinx/etc/example.sql</p></div>
</code></strong></p></li>
<li><p>Run the indexer to create full-text index from your data:</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;cd&nbsp;/usr/local/sphinx/etc<br>
$&nbsp;/usr/local/sphinx/bin/indexer&nbsp;--all</p></div>
</code></strong></p></li>
<li><p>Query your newly created index!</p></li>
</ol></div>
<p>
To query the index from command line, use <code class="filename">search</code> utility:
</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;cd&nbsp;/usr/local/sphinx/etc<br>
$&nbsp;/usr/local/sphinx/bin/search&nbsp;test</p></div>
</code></strong></p><p>
To query the index from your PHP scripts, you need to:
</p><div class="orderedlist"><ol type="1"><li><p>Run the search daemon which your script will talk to:</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;cd&nbsp;/usr/local/sphinx/etc<br>
$&nbsp;/usr/local/sphinx/bin/searchd</p></div>
</code></strong></p></li>
<li><p>
		Run the attached PHP API test script (to ensure that the daemon
		was succesfully started and is ready to serve the queries):
		</p><p><strong class="userinput"><code><div class="literallayout"><p>$&nbsp;cd&nbsp;sphinx/api<br>
$&nbsp;php&nbsp;test.php&nbsp;test</p></div>
</code></strong></p></li>
<li><p>
		Include the API (it's located in <code class="filename">api/sphinxapi.php</code>)
		into your own scripts and use it.
		</p></li>
</ol></div>
<p>
Happy searching!
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="indexing"></a>Chapter&nbsp;3.&nbsp;Indexing</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#sources">3.1. Data sources</a></span></dt>
<dt><span class="sect1"><a href="#attributes">3.2. Attributes</a></span></dt>
<dt><span class="sect1"><a href="#mva">3.3. MVA (multi-valued attributes)</a></span></dt>
<dt><span class="sect1"><a href="#indexes">3.4. Indexes</a></span></dt>
<dt><span class="sect1"><a href="#data-restrictions">3.5. Restrictions on the source data</a></span></dt>
<dt><span class="sect1"><a href="#charsets">3.6. Charsets, case folding, and translation tables</a></span></dt>
<dt><span class="sect1"><a href="#sql">3.7. SQL data sources (MySQL, PostgreSQL)</a></span></dt>
<dt><span class="sect1"><a href="#xmlpipe">3.8. xmlpipe data source</a></span></dt>
<dt><span class="sect1"><a href="#xmlpipe2">3.9. xmlpipe2 data source</a></span></dt>
<dt><span class="sect1"><a href="#live-updates">3.10. Live index updates</a></span></dt>
<dt><span class="sect1"><a href="#delta-updates">3.11. Delta index updates</a></span></dt>
<dt><span class="sect1"><a href="#index-merging">3.12. Index merging</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sources"></a>3.1.&nbsp;Data sources</h2></div></div></div>
<p>
The data to be indexed can generally come from very different
sources: SQL databases, plain text files, HTML files, mailboxes,
and so on. From Sphinx point of view, the data it indexes is a
set of structured <em class="glossterm">documents</em>, each of which has the
same set of <em class="glossterm">fields</em>. This is biased towards SQL, where
each row correspond to a document, and each column to a field.
</p><p>
Depending on what source Sphinx should get the data from,
different code is required to fetch the data and prepare it for indexing.
This code is called <em class="glossterm">data source driver</em> (or simply
<em class="glossterm">driver</em> or <em class="glossterm">data source</em> for brevity).
</p><p>
At the time of this writing, there are drivers for MySQL and
PostgreSQL databases, which can connect to the database using
its native C/C++ API, run queries and fetch the data. There's
also a driver called xmlpipe, which runs a specified command
and reads the data from its <code class="filename">stdout</code>.
See <a href="#xmlpipe" title="3.8.&nbsp;xmlpipe data source">Section&nbsp;3.8, &#8220;xmlpipe data source&#8221;</a> section for the format description.
</p><p>
There can be as many sources per index as necessary. They will be
sequentially processed in the very same order which was specifed in
index definition. All the documents coming from those sources
will be merged as if they were coming from a single source.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="attributes"></a>3.2.&nbsp;Attributes</h2></div></div></div>
<p>
Attributes are additional values associated with each document
that can be used to perform additional filtering and sorting during search.
</p><p>
It is often desired to additionally process full-text search results
based not only on matching document ID and its rank, but on a number
of other per-document values as well. For instance, one might need to
sort news search results by date and then relevance,
or search through products within specified price range,
or limit blog search to posts made by selected users,
or group results by month. To do that efficiently, Sphinx allows
to attach a number of additional <em class="glossterm">attributes</em>
to each document, and store their values in the full-text index.
It's then possible to use stored values to filter, sort,
or group full-text matches.
</p><p>Attributes, unlike the fields, are not full-text indexed. They
are stored in the index, but it is not possible to search them as full-text,
and attempting to do so results in an error.</p><p>For example, it is impossible to use the extended matching mode expression
<code class="option">@column 1</code> to match documents where column is 1, if column is an
attribute, and this is still true even if the numeric digits are normally indexed.</p><p>Attributes can be used for filtering, though, to restrict returned
rows, as well as sorting or <a href="#clustering" title="5.6.&nbsp;Grouping (clustering) search results ">result grouping</a>;
it is entirely possible to sort results purely based on attributes, and ignore the search
relevance tools. Additionally, attributes are returned from the search daemon, while the
indexed text is not.</p><p>
A good example for attributes would be a forum posts table. Assume
that only title and content fields need to be full-text searchable -
but that sometimes it is also required to limit search to a certain
author or a sub-forum (ie. search only those rows that have some
specific values of author_id or forum_id columns in the SQL table);
or to sort matches by post_date column; or to group matching posts
by month of the post_date and calculate per-group match counts.
</p><p>
This can be achieved by specifying all the mentioned columns
(excluding title and content, that are full-text fields) as
attributes, indexing them, and then using API calls to
setup filtering, sorting, and grouping. Here as an example.
</p><h3>Example sphinx.conf part:</h3><p>
</p><pre class="programlisting">
...
sql_query = SELECT id, title, content, \
	author_id, forum_id, post_date FROM my_forum_posts
sql_attr_uint = author_id
sql_attr_uint = forum_id
sql_attr_timestamp = post_date
...
</pre><p>
</p><h3>Example application code (in PHP):</h3><p>
</p><pre class="programlisting">
// only search posts by author whose ID is 123
$cl-&gt;SetFilter ( "author_id", array ( 123 ) );

// only search posts in sub-forums 1, 3 and 7
$cl-&gt;SetFilter ( "forum_id", array ( 1,3,7 ) );

// sort found posts by posting date in descending order
$cl-&gt;SetSortMode ( SPH_SORT_ATTR_DESC, "post_date" );
</pre><p>
</p><p>
Attributes are named. Attribute names are case insensitive.
Attributes are <span class="emphasis"><em>not</em></span> full-text indexed; they are stored in the index as is.
Currently supported attribute types are:
</p><div class="itemizedlist"><ul type="disc"><li>unsigned integers (1-bit to 32-bit wide);</li>
<li>UNIX timestamps;</li>
<li>floating point values (32-bit, IEEE 754 single precision);</li>
<li>string ordinals (specially computed integers);</li>
<li><a href="#conf-sql-attr-string" title="11.1.24.&nbsp;sql_attr_string">strings</a> (since 1.10-beta);</li>
<li><a href="#mva" title="3.3.&nbsp;MVA (multi-valued attributes)">MVA</a>, multi-value attributes (variable-length lists of 32-bit unsigned integers).</li>
</ul></div>
<p>
</p><p>
The complete set of per-document attribute values is sometimes
referred to as <em class="glossterm">docinfo</em>. Docinfos can either be
</p><div class="itemizedlist"><ul type="disc"><li>stored separately from the main full-text index data ("extern" storage, in <code class="filename">.spa</code> file), or</li>
<li>attached to each occurence of document ID in full-text index data ("inline" storage, in <code class="filename">.spd</code> file).</li>
</ul></div>
<p>
</p><p>
When using extern storage, a copy of <code class="filename">.spa</code> file
(with all the attribute values for all the documents) is kept in RAM by
<code class="filename">searchd</code> at all times. This is for performance reasons;
random disk I/O would be too slow. On the contrary, inline storage does not
require any additional RAM at all, but that comes at the cost of greatly
inflating the index size: remember that it copies <span class="emphasis"><em>all</em></span>
attribute value <span class="emphasis"><em>every</em></span> time when the document ID
is mentioned, and that is exactly as many times as there are
different keywords in the document. Inline may be the only viable
option if you have only a few attributes and need to work with big
datasets in limited RAM. However, in most cases extern storage
makes both indexing and searching <span class="emphasis"><em>much</em></span> more efficient.
</p><p>
Search-time memory requirements for extern storage are
(1+number_of_attrs)*number_of_docs*4 bytes, ie. 10 million docs with
2 groups and 1 timestamp will take (1+2+1)*10M*4 = 160 MB of RAM.
This is <span class="emphasis"><em>PER DAEMON</em></span>, not per query. <code class="filename">searchd</code>
will allocate 160 MB on startup, read the data and keep it shared between queries.
The children will <span class="emphasis"><em>NOT</em></span> allocate any additional
copies of this data.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="mva"></a>3.3.&nbsp;MVA (multi-valued attributes)</h2></div></div></div>
<p>
MVAs, or multi-valued attributes, are an important special type of per-document attributes in Sphinx.
MVAs make it possible to attach lists of values to every document.
They are useful for article tags, product categories, etc.
Filtering and group-by (but not sorting) on MVA attributes is supported.
</p><p>
Currently, MVA list entries are limited to unsigned 32-bit integers.
The list length is not limited, you can have an arbitrary number of values
attached to each document as long as RAM permits (<code class="filename">.spm</code> file
that contains the MVA values will be precached in RAM by <code class="filename">searchd</code>).
The source data can be taken either from a separate query, or from a document field;
see source type in <a href="#conf-sql-attr-multi" title="11.1.23.&nbsp;sql_attr_multi">sql_attr_multi</a>.
In the first case the query will have to return pairs of document ID and MVA values,
in the second one the field will be parsed for integer values.
There are absolutely no requirements as to incoming data order; the values will be
automatically grouped by document ID (and internally sorted within the same ID)
during indexing anyway.
</p><p>
When filtering, a document will match the filter on MVA attribute
if <span class="emphasis"><em>any</em></span> of the values satisfy the filtering condition.
(Therefore, documents that pass through exclude filters will not
contain any of the forbidden values.)
When grouping by MVA attribute, a document will contribute to as
many groups as there are different MVA values associated with that document.
For instance, if the collection contains exactly 1 document having a 'tag' MVA
with values 5, 7, and 11, grouping on 'tag' will produce 3 groups with
'@count' equal to 1 and '@groupby' key values of 5, 7, and 11 respectively.
Also note that grouping by MVA might lead to duplicate documents in the result set:
because each document can participate in many groups, it can be chosen as the best
one in in more than one group, leading to duplicate IDs. PHP API historically
uses ordered hash on the document ID for the resulting rows; so you'll also need to use
<a href="#api-func-setarrayresult" title="8.1.6.&nbsp;SetArrayResult">SetArrayResult()</a> in order
to employ group-by on MVA with PHP API.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="indexes"></a>3.4.&nbsp;Indexes</h2></div></div></div>
<p>
To be able to answer full-text search queries fast, Sphinx needs
to build a special data structure optimized for such queries from
your text data. This structure is called <em class="glossterm">index</em>; and
the process of building index from text is called <em class="glossterm">indexing</em>.
</p><p>
Different index types are well suited for different tasks.
For example, a disk-based tree-based index would be easy to
update (ie. insert new documents to existing index), but rather
slow to search. Therefore, Sphinx architecture allows for different
<em class="glossterm">index types</em> to be implemented easily.
</p><p>
The only index type which is implemented in Sphinx at the moment is
designed for maximum indexing and searching speed. This comes at a cost
of updates being really slow; theoretically, it might be slower to
update this type of index than than to reindex it from scratch.
However, this very frequently could be worked around with
muiltiple indexes, see <a href="#live-updates" title="3.10.&nbsp;Live index updates">Section&nbsp;3.10, &#8220;Live index updates&#8221;</a> for details.
</p><p>
It is planned to implement more index types, including the
type which would be updateable in real time.
</p><p>
There can be as many indexes per configuration file as necessary.
<code class="filename">indexer</code> utility can reindex either all of them
(if <code class="option">--all</code> option is specified), or a certain explicitly
specified subset. <code class="filename">searchd</code> utility will serve all
the specified indexes, and the clients can specify what indexes to
search in run time.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="data-restrictions"></a>3.5.&nbsp;Restrictions on the source data</h2></div></div></div>
<p>
There are a few different restrictions imposed on the source data
which is going to be indexed by Sphinx, of which the single most
important one is:
</p><p><span class="bold"><strong>
ALL DOCUMENT IDS MUST BE UNIQUE UNSIGNED NON-ZERO INTEGER NUMBERS (32-BIT OR 64-BIT, DEPENDING ON BUILD TIME SETTINGS).
</strong></span></p><p>
If this requirement is not met, different bad things can happen.
For instance, Sphinx can crash with an internal assertion while indexing;
or produce strange results when searching due to conflicting IDs.
Also, a 1000-pound gorilla might eventually come out of your
display and start throwing barrels at you. You've been warned.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="charsets"></a>3.6.&nbsp;Charsets, case folding, and translation tables</h2></div></div></div>
<p>
When indexing some index, Sphinx fetches documents from
the specified sources, splits the text into words, and does
case folding so that "Abc", "ABC" and "abc" would be treated
as the same word (or, to be pedantic, <em class="glossterm">term</em>).
</p><p>
To do that properly, Sphinx needs to know
</p><div class="itemizedlist"><ul type="disc"><li>what encoding is the source text in;</li>
<li>what characters are letters and what are not;</li>
<li>what letters should be folded to what letters.</li>
</ul></div>
<p>
This should be configured on a per-index basis using
<code class="option"><a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a></code> and 
<code class="option"><a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a></code> options.
<code class="option"><a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a></code>
specifies whether the document encoding is single-byte (SBCS) or UTF-8.
<code class="option"><a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a></code>
specifies the table that maps letter characters to their case
folded versions. The characters that are not in the table are considered
to be non-letters and will be treated as word separators when indexing
or searching through this index.
</p><p>
Note that while default tables do not include space character
(ASCII code 0x20, Unicode U+0020) as a letter, it's in fact
<span class="emphasis"><em>perfectly legal</em></span> to do so. This can be
useful, for instance, for indexing tag clouds, so that space-separated
word sets would index as a <span class="emphasis"><em>single</em></span> search query term.
</p><p>
Default tables currently include English and Russian characters.
Please do submit your tables for other languages!
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sql"></a>3.7.&nbsp;SQL data sources (MySQL, PostgreSQL)</h2></div></div></div>
<p>
With all the SQL drivers, indexing generally works as follows.
</p><div class="itemizedlist"><ul type="disc"><li>connection to the database is established;</li>
<li>pre-query (see <a href="#conf-sql-query-pre" title="11.1.11.&nbsp;sql_query_pre">Section&nbsp;11.1.11, &#8220;sql_query_pre&#8221;</a>) is executed
	to perform any necessary initial setup, such as setting per-connection encoding with MySQL;</li>
<li>main query (see <a href="#conf-sql-query" title="11.1.12.&nbsp;sql_query">Section&nbsp;11.1.12, &#8220;sql_query&#8221;</a>) is executed and the rows it returns are indexed;</li>
<li>post-query (see <a href="#conf-sql-query-post" title="11.1.29.&nbsp;sql_query_post">Section&nbsp;11.1.29, &#8220;sql_query_post&#8221;</a>) is executed
	to perform any necessary cleanup;</li>
<li>connection to the database is closed;</li>
<li>indexer does the sorting phase (to be pedantic, index-type specific post-processing);</li>
<li>connection to the database is established again;</li>
<li>post-index query (see <a href="#conf-sql-query-post-index" title="11.1.30.&nbsp;sql_query_post_index">Section&nbsp;11.1.30, &#8220;sql_query_post_index&#8221;</a>) is executed
	to perform any necessary final cleanup;</li>
<li>connection to the database is closed again.</li>
</ul></div>
<p>
Most options, such as database user/host/password, are straightforward.
However, there are a few subtle things, which are discussed in more detail here.
</p><h3><a name="ranged-queries"></a>Ranged queries</h3><p>
Main query, which needs to fetch all the documents, can impose
a read lock on the whole table and stall the concurrent queries
(eg. INSERTs to MyISAM table), waste a lot of memory for result set, etc.
To avoid this, Sphinx supports so-called <em class="glossterm">ranged queries</em>.
With ranged queries, Sphinx first fetches min and max document IDs from
the table, and then substitutes different ID intervals into main query text
and runs the modified query to fetch another chunk of documents.
Here's an example.
</p><div class="example"><a name="ex-ranged-queries"></a><p class="title"><b>Example&nbsp;3.1.&nbsp;Ranged query usage example</b></p><div class="example-contents"><pre class="programlisting">
# in sphinx.conf

sql_query_range	= SELECT MIN(id),MAX(id) FROM documents
sql_range_step = 1000
sql_query = SELECT * FROM documents WHERE id&gt;=$start AND id&lt;=$end
</pre></div></div>
<br class="example-break"><p>
If the table contains document IDs from 1 to, say, 2345, then sql_query would
be run three times:
</p><div class="orderedlist"><ol type="1"><li>with <code class="option">$start</code> replaced with 1 and <code class="option">$end</code> replaced with 1000;</li>
<li>with <code class="option">$start</code> replaced with 1001 and <code class="option">$end</code> replaced with 2000;</li>
<li>with <code class="option">$start</code> replaced with 2000 and <code class="option">$end</code> replaced with 2345.</li>
</ol></div>
<p>
Obviously, that's not much of a difference for 2000-row table,
but when it comes to indexing 10-million-row MyISAM table,
ranged queries might be of some help.
</p><h3><code class="option">sql_post</code> vs. <code class="option">sql_post_index</code></h3><p>
The difference between post-query and post-index query is in that post-query
is run immediately when Sphinx received all the documents, but further indexing
<span class="bold"><strong>may</strong></span> still fail for some other reason. On the contrary,
by the time the post-index query gets executed, it is <span class="bold"><strong>guaranteed</strong></span>
that the indexing was succesful. Database connection is dropped and re-established
because sorting phase can be very lengthy and would just timeout otherwise.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="xmlpipe"></a>3.8.&nbsp;xmlpipe data source</h2></div></div></div>
<p>
xmlpipe data source was designed to enable users to plug data into
Sphinx without having to implement new data sources drivers themselves.
It is limited to 2 fixed fields and 2 fixed attributes, and is deprecated
in favor of <a href="#xmlpipe2" title="3.9.&nbsp;xmlpipe2 data source">Section&nbsp;3.9, &#8220;xmlpipe2 data source&#8221;</a> now. For new streams, use xmlpipe2.
</p><p>
To use xmlpipe, configure the data source in your configuration file
as follows:
</p><pre class="programlisting">
source example_xmlpipe_source
{
    type = xmlpipe
    xmlpipe_command = perl /www/mysite.com/bin/sphinxpipe.pl
}
</pre><p>
The <code class="filename">indexer</code> will run the command specified
in <code class="option"><a href="#conf-xmlpipe-command" title="11.1.33.&nbsp;xmlpipe_command">xmlpipe_command</a></code>,
and then read, parse and index the data it prints to <code class="filename">stdout</code>.
More formally, it opens a pipe to given command and then reads
from that pipe.
</p><p>
indexer will expect one or more documents in custom XML format.
Here's the example document stream, consisting of two documents:
</p><div class="example"><a name="ex-xmlpipe-document"></a><p class="title"><b>Example&nbsp;3.2.&nbsp;XMLpipe document stream</b></p><div class="example-contents"><pre class="programlisting">
&lt;document&gt;
&lt;id&gt;123&lt;/id&gt;
&lt;group&gt;45&lt;/group&gt;
&lt;timestamp&gt;1132223498&lt;/timestamp&gt;
&lt;title&gt;test title&lt;/title&gt;
&lt;body&gt;
this is my document body
&lt;/body&gt;
&lt;/document&gt;

&lt;document&gt;
&lt;id&gt;124&lt;/id&gt;
&lt;group&gt;46&lt;/group&gt;
&lt;timestamp&gt;1132223498&lt;/timestamp&gt;
&lt;title&gt;another test&lt;/title&gt;
&lt;body&gt;
this is another document
&lt;/body&gt;
&lt;/document&gt;
</pre></div></div>
<p><br class="example-break">
</p><p>
Legacy xmlpipe legacy driver uses a builtin parser
which is pretty fast but really strict and does not actually
fully support XML. It requires that all the fields <span class="emphasis"><em>must</em></span>
be present, formatted <span class="emphasis"><em>exactly</em></span> as in this example, and
occur <span class="emphasis"><em>exactly</em></span> in the same order. The only optional
field is <code class="option">timestamp</code>; it defaults to 1.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="xmlpipe2"></a>3.9.&nbsp;xmlpipe2 data source</h2></div></div></div>
<p>
xmlpipe2 lets you pass arbitrary full-text and attribute data to Sphinx
in yet another custom XML format. It also allows to specify the schema
(ie. the set of fields and attributes) either in the XML stream itself,
or in the source settings.
</p><p>
When indexing xmlpipe2 source, indexer runs the given command, opens
a pipe to its stdout, and expects well-formed XML stream. Here's sample
stream data:
</p><div class="example"><a name="ex-xmlpipe2-document"></a><p class="title"><b>Example&nbsp;3.3.&nbsp;xmlpipe2 document stream</b></p><div class="example-contents"><pre class="programlisting">
&lt;?xml version="1.0" encoding="utf-8"?&gt;
&lt;sphinx:docset&gt;

&lt;sphinx:schema&gt;
&lt;sphinx:field name="subject"/&gt; 
&lt;sphinx:field name="content"/&gt;
&lt;sphinx:attr name="published" type="timestamp"/&gt;
&lt;sphinx:attr name="author_id" type="int" bits="16" default="1"/&gt;
&lt;/sphinx:schema&gt;

&lt;sphinx:document id="1234"&gt;
&lt;content&gt;this is the main content &lt;![CDATA[[and this &lt;cdata&gt; entry
must be handled properly by xml parser lib]]&gt;&lt;/content&gt;
&lt;published&gt;1012325463&lt;/published&gt;
&lt;subject&gt;note how field/attr tags can be
in &lt;b class="red"&gt;randomized&lt;/b&gt; order&lt;/subject&gt;
&lt;misc&gt;some undeclared element&lt;/misc&gt;
&lt;/sphinx:document&gt;

&lt;sphinx:document id="1235"&gt;
&lt;subject&gt;another subject&lt;/subject&gt;
&lt;content&gt;here comes another document, and i am given to understand,
that in-document field order must not matter, sir&lt;/content&gt;
&lt;published&gt;1012325467&lt;/published&gt;
&lt;/sphinx:document&gt;

&lt;!-- ... even more sphinx:document entries here ... --&gt;

&lt;sphinx:killlist&gt;
&lt;id&gt;1234&lt;/id&gt;
&lt;id&gt;4567&lt;/id&gt;
&lt;/sphinx:killlist&gt;

&lt;/sphinx:docset&gt;
</pre></div></div>
<p><br class="example-break">
</p><p>
Arbitrary fields and attributes are allowed.
They also can occur in the stream in arbitrary order within each document; the order is ignored.
There is a restriction on maximum field length; fields longer than 2 MB will be truncated to 2 MB (this limit can be changed in the source).
</p><p>
The schema, ie. complete fields and attributes list, must be declared
before any document could be parsed. This can be done either in the
configuration file using <code class="option">xmlpipe_field</code> and <code class="option">xmlpipe_attr_XXX</code>
settings, or right in the stream using &lt;sphinx:schema&gt; element.
&lt;sphinx:schema&gt; is optional. It is only allowed to occur as the very
first sub-element in &lt;sphinx:docset&gt;. If there is no in-stream
schema definition, settings from the configuration file will be used.
Otherwise, stream settings take precedence.
</p><p>
Unknown tags (which were not declared neither as fields nor as attributes)
will be ignored with a warning. In the example above, &lt;misc&gt; will be ignored.
All embedded tags and their attributes (such as &lt;b&gt; in &lt;subject&gt;
in the example above) will be silently ignored.
</p><p>
Support for incoming stream encodings depends on whether <code class="filename">iconv</code>
is installed on the system. xmlpipe2 is parsed using <code class="filename">libexpat</code>
parser that understands US-ASCII, ISO-8859-1, UTF-8 and a few UTF-16 variants
natively. Sphinx <code class="filename">configure</code> script will also check
for <code class="filename">libiconv</code> presence, and utilize it to handle
other encodings. <code class="filename">libexpat</code> also enforces the
requirement to use UTF-8 charset on Sphinx side, because the
parsed data it returns is always in UTF-8.

</p><p>
XML elements (tags) recognized by xmlpipe2 (and their attributes where applicable) are:
</p><div class="variablelist"><dl><dt><span class="term">sphinx:docset</span></dt>
<dd>Mandatory top-level element, denotes and contains xmlpipe2 document set.</dd><dt><span class="term">sphinx:schema</span></dt>
<dd>Optional element, must either occur as the very first child
		of sphinx:docset, or never occur at all. Declares the document schema.
		Contains field and attribute declarations. If present, overrides
		per-source settings from the configuration file.
	</dd><dt><span class="term">sphinx:field</span></dt>
<dd>Optional element, child of sphinx:schema. Declares a full-text field.
		Known attributes are:
		<div class="itemizedlist"><ul type="disc"><li>"name", specifies the XML element name that will be treated as a full-text field in the subsequent documents.</li>
<li>"attr", specifies whether to also index this field as a string or word count attribute. Possible values are "string" and "wordcount". Introduced in version 1.10-beta.</li>
</ul></div>
</dd><dt><span class="term">sphinx:attr</span></dt>
<dd>Optional element, child of sphinx:schema. Declares an attribute.
		Known attributes are:
		<div class="itemizedlist"><ul type="disc"><li>"name", specifies the element name that should be treated as an attribute in the subsequent documents.</li>
<li>"type", specifies the attribute type. Possible values are "int", "timestamp", "str2ordinal", "bool", "float" and "multi".</li>
<li>"bits", specifies the bit size for "int" attribute type. Valid values are 1 to 32.</li>
<li>"default", specifies the default value for this attribute that should be used if the attribute's element is not present in the document.</li>
</ul></div>
</dd><dt><span class="term">sphinx:document</span></dt>
<dd>Mandatory element, must be a child of sphinx:docset.
		Contains arbitrary other elements with field and attribute values
		to be indexed, as declared either using sphinx:field and sphinx:attr
		elements or in the configuration file. The only known attribute
		is "id" that must contain the unique integer document ID.
	</dd><dt><span class="term">sphinx:killlist</span></dt>
<dd>Optional element, child of sphinx:docset.
		Contains a number of "id" elements whose contents are document IDs
		to be put into a <a href="#conf-sql-query-killlist" title="11.1.16.&nbsp;sql_query_killlist">kill-list</a> for this index.
	</dd></dl></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="live-updates"></a>3.10.&nbsp;Live index updates</h2></div></div></div>
<p>
There are two major approaches to maintaining the full-text index
contents up to date. Note, however, that both these approaches deal
with the task of <span class="emphasis"><em>full-text data updates</em></span>, and not
attribute updates. Instant attribute updates are supported since 
version 0.9.8. Refer to <a href="#api-func-updateatttributes" title="8.7.2.&nbsp;UpdateAttributes">UpdateAttributes()</a>
API call description for details.
</p><p>
First, you can use disk-based indexes, partition them manually,
and only rebuild the smaller partitions (so-called "deltas") frequently.
By minimizing the rebuild size, you can reduce the average indexing lag
to something as low as 30-60 seconds. This approach was the the only one
available in versions 0.9.x. On huge collections it actually might be
the most efficient one. Refer to <a href="#delta-updates" title="3.11.&nbsp;Delta index updates">Section&nbsp;3.11, &#8220;Delta index updates&#8221;</a>
for details.
</p><p>
Second, versions 1.x (starting with 1.10-beta) add support for so-called
real-time indexes (RT indexes for short) that on-the-fly updates of the
full-text data. Updates on a RT index can appear in the search results in 
1-2 milliseconds, ie. 0.001-0.002 seconds. However, RT index are less 
efficient for bulk indexing huge amounts of data. Refer to 
<a href="#rt-indexes" title="Chapter&nbsp;4.&nbsp;Real-time indexes">Chapter&nbsp;4, <i>Real-time indexes</i></a> for details.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="delta-updates"></a>3.11.&nbsp;Delta index updates</h2></div></div></div>
<p>
There's a frequent situation when the total dataset is too big
to be reindexed from scratch often, but the amount of new records
is rather small. Example: a forum with a 1,000,000 archived posts,
but only 1,000 new posts per day.
</p><p>
In this case, "live" (almost real time) index updates could be
implemented using so called "main+delta" scheme.
</p><p>
The idea is to set up two sources and two indexes, with one
"main" index for the data which only changes rarely (if ever),
and one "delta" for the new documents. In the example above,
1,000,000 archived posts would go to the main index, and newly
inserted 1,000 posts/day would go to the delta index. Delta index
could then be reindexed very frequently, and the documents can
be made available to search in a matter of minutes.
</p><p>
Specifying which documents should go to what index and
reindexing main index could also be made fully automatical.
One option would be to make a counter table which would track
the ID which would split the documents, and update it
whenever the main index is reindexed.
</p><div class="example"><a name="ex-live-updates"></a><p class="title"><b>Example&nbsp;3.4.&nbsp;Fully automated live updates</b></p><div class="example-contents"><pre class="programlisting">
# in MySQL
CREATE TABLE sph_counter
(
    counter_id INTEGER PRIMARY KEY NOT NULL,
    max_doc_id INTEGER NOT NULL
);

# in sphinx.conf
source main
{
    # ...
    sql_query_pre = SET NAMES utf8
    sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX(id) FROM documents
    sql_query = SELECT id, title, body FROM documents \
        WHERE id&lt;=( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}

source delta : main
{
    sql_query_pre = SET NAMES utf8
    sql_query = SELECT id, title, body FROM documents \
        WHERE id&gt;( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}

index main
{
    source = main
    path = /path/to/main
    # ... all the other settings
}

# note how all other settings are copied from main,
# but source and path are overridden (they MUST be)
index delta : main
{
    source = delta
    path = /path/to/delta
}
</pre></div></div>
<p><br class="example-break">
</p><p>
Note how we're overriding <code class="code">sql_query_pre</code> in the delta source.
We need to explicitly have that override. Otherwise <code class="code">REPLACE</code> query
would be run when indexing delta source too, effectively nullifying it. However,
when we issue the directive in the inherited source for the first time, it removes 
<span class="emphasis"><em>all</em></span> inherited values, so the encoding setup is also lost.
So <code class="code">sql_query_pre</code> in the delta can not just be empty; and we need
to issue the encoding setup query explicitly once again.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="index-merging"></a>3.12.&nbsp;Index merging</h2></div></div></div>
<p>
Merging two existing indexes can be more efficient that indexing the data
from scratch, and desired in some cases (such as merging 'main' and 'delta'
indexes instead of simply reindexing 'main' in 'main+delta' partitioning
scheme). So <code class="filename">indexer</code> has an option to do that.
Merging the indexes is normally faster than reindexing but still
<span class="emphasis"><em>not</em></span> instant on huge indexes. Basically,
it will need to read the contents of both indexes once and write
the result once. Merging 100 GB and 1 GB index, for example,
will result in 202 GB of IO (but that's still likely less than
the indexing from scratch requires).
</p><p>
The basic command syntax is as follows:
</p><pre class="programlisting">
indexer --merge DSTINDEX SRCINDEX [--rotate]
</pre><p>
Only the DSTINDEX index will be affected: the contents of SRCINDEX will be merged into it.
<code class="option">--rotate</code> switch will be required if DSTINDEX is already being served by <code class="filename">searchd</code>.
The initially devised usage pattern is to merge a smaller update from SRCINDEX into DSTINDEX.
Thus, when merging the attributes, values from SRCINDEX will win if duplicate document IDs are encountered.
Note, however, that the "old" keywords will <span class="emphasis"><em>not</em></span> be automatically removed in such cases.
For example, if there's a keyword "old" associated with document 123 in DSTINDEX, and a keyword "new" associated
with it in SRCINDEX, document 123 will be found by <span class="emphasis"><em>both</em></span> keywords after the merge.
You can supply an explicit condition to remove documents from DSTINDEX to mitigate that;
the relevant switch is <code class="option">--merge-dst-range</code>:
</p><pre class="programlisting">
indexer --merge main delta --merge-dst-range deleted 0 0
</pre><p>
This switch lets you apply filters to the destination index along with merging.
There can be several filters; all of their conditions must be met in order
to include the document in the resulting mergid index. In the example above,
the filter passes only those records where 'deleted' is 0, eliminating all
records that were flagged as deleted (for instance, using
<a href="#api-func-updateatttributes" title="8.7.2.&nbsp;UpdateAttributes">UpdateAttributes()</a> call).
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="rt-indexes"></a>Chapter&nbsp;4.&nbsp;Real-time indexes</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#rt-overview">4.1. RT indexes overview</a></span></dt>
<dt><span class="sect1"><a href="#rt-caveats">4.2. Known caveats with RT indexes</a></span></dt>
<dt><span class="sect1"><a href="#rt-internals">4.3. RT index internals</a></span></dt>
<dt><span class="sect1"><a href="#rt-binlog">4.4. Binary logging</a></span></dt>
</dl></div>
<p>
Real-time indexes (or RT indexes for brevity) are a new backend
that lets you insert, update, or delete documents (rows) on the fly.
RT indexes were added in version 1.10-beta.  While querying of RT indexes
is possible using any of the SphinxAPI, SphinxQL, or SphinxSE, updating
them is only possible via SphinxQL at the moment.  Full SphinxQL
reference is available in <a href="#sphinxql-reference" title="Chapter&nbsp;7.&nbsp;SphinxQL reference">Chapter&nbsp;7, <i>SphinxQL reference</i></a>.
</p><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rt-overview"></a>4.1.&nbsp;RT indexes overview</h2></div></div></div>
<p>
RT indexes should be declared in <code class="filename">sphinx.conf</code>,
just as every other index type. Notable differences from the regular, 
disk-based indexes are that a) data sources are not required and ignored,
and b) you should explicitly enumerate all the text fields, not just
attributes. Here's an example:
</p><div class="example"><a name="ex-rt-updates"></a><p class="title"><b>Example&nbsp;4.1.&nbsp;RT index declaration</b></p><div class="example-contents"><pre class="programlisting">
index rt
{
	type = rt
	path = /usr/local/sphinx/data/rt
	rt_field = title
	rt_field = content
	rt_attr_uint = gid
}
</pre></div></div>
<br class="example-break"><p>
RT INDEXES ARE CURRENTLY (AS OF VERSION 1.10-beta) A WORK IN PROGRESS.
Therefore, they might lack certain features: for instance, prefix/infix
indexing, MVA attributes, etc are not supported yet.  There also might be
performance and stability issues.  However, all the regular indexing features
and most of the searching features are already in place, our internal
testing passes, and last but not least a number of production instances
are already using RT indexes with good results.
</p><p>
RT index can be accessed using MySQL protocol. INSERT, DELETE, and
SELECT statements against RT index are supported. For instance, this
is an example session with the sample index above:
</p><pre class="programlisting">
$ mysql -h 127.0.0.1 -P 9306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 1.10-dev (r2153)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql&gt; INSERT INTO rt VALUES ( 1, 'first record', 'test one', 123 );
Query OK, 1 row affected (0.05 sec)

mysql&gt; INSERT INTO rt VALUES ( 2, 'second record', 'test two', 234 );
Query OK, 1 row affected (0.00 sec)

mysql&gt; SELECT * FROM rt;
+------+--------+------+
| id   | weight | gid  |
+------+--------+------+
|    1 |      1 |  123 |
|    2 |      1 |  234 |
+------+--------+------+
2 rows in set (0.02 sec)

mysql&gt; SELECT * FROM rt WHERE MATCH('test');
+------+--------+------+
| id   | weight | gid  |
+------+--------+------+
|    1 |   1643 |  123 |
|    2 |   1643 |  234 |
+------+--------+------+
2 rows in set (0.01 sec)

mysql&gt; SELECT * FROM rt WHERE MATCH('@title test');
Empty set (0.00 sec)
</pre><p>
Both partial and batch INSERT syntaxes are supported, ie. 
you can specify a subset of columns, and insert several rows at a time. 
Deletions are also possible using DELETE statement; the only currently
supported syntax is DELETE FROM &lt;index&gt; WHERE id=&lt;id&gt;.
REPLACE is also supported, enabling you to implement updates.
</p><pre class="programlisting">
mysql&gt; INSERT INTO rt ( id, title ) VALUES ( 3, 'third row' ), ( 4, 'fourth entry' );
Query OK, 2 rows affected (0.01 sec)

mysql&gt; SELECT * FROM rt;
+------+--------+------+
| id   | weight | gid  |
+------+--------+------+
|    1 |      1 |  123 |
|    2 |      1 |  234 |
|    3 |      1 |    0 |
|    4 |      1 |    0 |
+------+--------+------+
4 rows in set (0.00 sec)

mysql&gt; DELETE FROM rt WHERE id=2;
Query OK, 0 rows affected (0.00 sec)

mysql&gt; SELECT * FROM rt WHERE MATCH('test');
+------+--------+------+
| id   | weight | gid  |
+------+--------+------+
|    1 |   1500 |  123 |
+------+--------+------+
1 row in set (0.00 sec)

mysql&gt; INSERT INTO rt VALUES ( 1, 'first record on steroids', 'test one', 123 );
ERROR 1064 (42000): duplicate id '1'

mysql&gt; REPLACE INTO rt VALUES ( 1, 'first record on steroids', 'test one', 123 );
Query OK, 1 row affected (0.01 sec)

mysql&gt; SELECT * FROM rt WHERE MATCH('steroids');
+------+--------+------+
| id   | weight | gid  |
+------+--------+------+
|    1 |   1500 |  123 |
+------+--------+------+
1 row in set (0.01 sec)
</pre><p>
Data stored in RT index should survive clean shutdown. When binary logging
is enabled, it should also survive crash and/or dirty shutdown, and recover
on subsequent startup.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rt-caveats"></a>4.2.&nbsp;Known caveats with RT indexes</h2></div></div></div>
<p>
As of 1.10-beta, RT indexes are a beta quality feature: while no major,
showstopper-class issues are known, there still are a few known usage quirks.
Those quirks are listed in this section.
</p><div class="itemizedlist"><ul type="disc"><li>Prefix and infix indexing are not supported yet.</li>
<li>MVAs are not supported yet.</li>
<li>Disk chunks optimization routine is not implemented yet.</li>
<li>On initial index creation, attributes are reordered by type,
in the following order: uint, bigint, float, timestamp, string. So when
using INSERT without an explicit column names list, specify all uint
column values first, then bigint, etc.</li>
<li>Default conservative RAM chunk limit (<code class="option">rt_mem_limit</code>)
of 32M can lead to poor performance on bigger indexes, you should raise it to
256..1024M if you're planning to index gigabytes.</li>
<li>High DELETE/REPLACE rate can lead to kill-list fragmentation
and impact searching performance.</li>
<li>No transaction size limits are currently imposed;
too many concurrent INSERT/REPLACE transactions might therefore
consume a lot of RAM.</li>
<li>In case of a damaged binlog, recovery will stop on the
first damaged transaction, even though it's technically possible
to keep looking further for subsequent undamaged transactions, and
recover those. This mid-file damage case (due to flaky HDD/CDD/tape?)
is supposed to be extremely rare, though.</li>
<li>Multiple INSERTs grouped in a single transaction perform
better than equivalent single-row transactions and are recommended for
batch loading of data.</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rt-internals"></a>4.3.&nbsp;RT index internals</h2></div></div></div>
<p>
RT index is internally chunked.  It keeps a so-called RAM chunk
that stores all the most recent changes.  RAM chunk memory usage
is rather strictly limited with per-index
<a href="#conf-rt-mem-limit" title="11.2.45.&nbsp;rt_mem_limit">rt_mem_limit</a> directive.
Once RAM chunk grows over this limit, a new disk chunk is created
from its data, and RAM chunk is reset.  Thus, while most changes
on the RT index will be performed in RAM only and complete instantly
(in milliseconds), those changes that overflow the RAM chunk will
stall for the duration of disk chunk creation (a few seconds).
</p><p>
Disk chunks are, in fact, just regular disk-based indexes.
But they're a part of an RT index and automatically managed by it,
so you need not configure nor manage them manually.  Because a new
disk chunk is created every time RT chunk overflows the limit, and
because in-memory chunk format is close to on-disk format, the disk
chunks will be approximately <code class="option">rt_mem_limit</code> bytes
in size each.
</p><p>
Generally, it is better to set the limit bigger, to minimize both
the frequency of flushes, and the index fragmentation (number of disk
chunks).  For instance, on a dedicated search server that handles
a big RT index, it can be advised to set <code class="option">rt_mem_limit</code>
to 1-3 GB.  A global limit on all indexes is also planned, but not yet
implemented yet as of 1.10-beta.
</p><p>
Disk chunk full-text index data can not be actually modified,
so the full-text field changes (ie. row deletions and updates)
suppress a previous row version from a disk chunk using a kill-list,
but do not actually physicall purge the data.  Therefore, on workloads
with high full-text updates ratio index might eventually get polluted
by these previous row versions, and searching performance would
degrade.  Physical index purging that would improve the performance
is planned, but not yet implemented as of 1.10-beta.
</p><p>
Data in RAM chunk gets saved to disk on clean daemon shutdown, and
then loaded back on startup.  However, on daemon or server crash,
updates from RAM chunk might be lost.  To prevent that, binary logging
of transactions can be used; see <a href="#rt-binlog" title="4.4.&nbsp;Binary logging">Section&nbsp;4.4, &#8220;Binary logging&#8221;</a> for details.
</p><p>
Full-text changes in RT index are transactional.  They are stored
in a per-thread accumulator until COMMIT, then applied at once.
Bigger batches per single COMMIT should result in faster indexing.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rt-binlog"></a>4.4.&nbsp;Binary logging</h2></div></div></div>
<p>
Binary logs are essentially a recovery mechanism.  With binary logs
enabled, <code class="filename">searchd</code> writes every given transaction
to the binlog file, and uses that for recovery after an unclean shutdown.
On clean shutdown, RAM chunks are saved to disk, and then all the binlog
files are unlinked.
</p><p>
During normal operation, a new binlog file will be opened every time
when <code class="option">binlog_max_log_size</code> limit (which defaults to 128M)
is reached.  Older, already closed binlog files are kept until all of the
transactions stored in them (from all indexes) are flushed as a disk chunk.
Setting the limit to 0 pretty much prevents binlog from being unlinked
at all while <code class="filename">searchd</code> is running; however, it will
still be unlinked on clean shutdown.
</p><p>
There are 3 different binlog flushing strategies, controlled by
<a href="#conf-binlog-flush" title="11.4.30.&nbsp;binlog_flush">binlog_flush</a> directive
which takes the values of 0, 1, or 2. 0 means to flush the log
to OS and sync it to disk every second; 1 means flush and sync
every transaction; and 2 (the default mode) means flush every
transaction but sync every second. Sync is relatively slow because
it has to perform physical disk writes, so mode 1 is the safest
(every committed transaction is guaranteed to be written on disk)
but the slowest. Flushing log to OS prevents from data loss on
<code class="filename">searchd</code> crashes but not system crashes.
Mode 2 is the default.
</p><p>
On recovery after an unclean shutdown, binlogs are replayed
and all logged transactions since the last good on-disk state
are restored. Transactions are checksummed so in case of binlog
file corruption garbage data will <span class="bold"><strong>not</strong></span> be replayed; such
a broken transaction will be detected and, currently, will stop
replay. Transactions also start with a magic marker and timestamped,
so in case of binlog damage in the middle of the file, it's technically
possible to skip broken transactions and keep replaying from the next
good one, and/or it's possible to replay transactions until a given
timestamp (point-in-time recovery), but none of that is implemented yet
as of 1.10-beta.
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="searching"></a>Chapter&nbsp;5.&nbsp;Searching</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#matching-modes">5.1. Matching modes</a></span></dt>
<dt><span class="sect1"><a href="#boolean-syntax">5.2. Boolean query syntax</a></span></dt>
<dt><span class="sect1"><a href="#extended-syntax">5.3. Extended query syntax</a></span></dt>
<dt><span class="sect1"><a href="#weighting">5.4. Weighting</a></span></dt>
<dt><span class="sect1"><a href="#sorting-modes">5.5. Sorting modes</a></span></dt>
<dt><span class="sect1"><a href="#clustering">5.6. Grouping (clustering) search results </a></span></dt>
<dt><span class="sect1"><a href="#distributed">5.7. Distributed searching</a></span></dt>
<dt><span class="sect1"><a href="#query-log-format">5.8. <code class="filename">searchd</code> query log format</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql">5.9. MySQL protocol support and SphinxQL</a></span></dt>
<dt><span class="sect1"><a href="#multi-queries">5.10. Multi-queries</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="matching-modes"></a>5.1.&nbsp;Matching modes</h2></div></div></div>
<p>
There are the following matching modes available:
</p><div class="itemizedlist"><ul type="disc"><li>SPH_MATCH_ALL, matches all query words (default mode);</li>
<li>SPH_MATCH_ANY, matches any of the query words;</li>
<li>SPH_MATCH_PHRASE, matches query as a phrase, requiring perfect match;</li>
<li>SPH_MATCH_BOOLEAN, matches query as a boolean expression (see <a href="#boolean-syntax" title="5.2.&nbsp;Boolean query syntax">Section&nbsp;5.2, &#8220;Boolean query syntax&#8221;</a>);</li>
<li>SPH_MATCH_EXTENDED, matches query as an expression in Sphinx internal query language
	(see <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">Section&nbsp;5.3, &#8220;Extended query syntax&#8221;</a>). As of 0.9.9, this has been superceded by SPH_MATCH_EXTENDED2,
	providing additional functionality and better performance. The ident is retained for legacy application code
	that will continue to be compatible once Sphinx and its components, including the API, are upgraded.</li>
<li>SPH_MATCH_EXTENDED2, matches query using the second version of the Extended matching mode.</li>
<li>SPH_MATCH_FULLSCAN, matches query, forcibly using the "full scan" mode as below.
	NB, any query terms will be ignored, such that filters, filter-ranges and grouping 
	will still be applied, but no text-matching.</li>
</ul></div>
<p>
</p><p>
The SPH_MATCH_FULLSCAN mode will be automatically activated in place of the specified matching mode when the following conditions are met:
</p><div class="orderedlist"><ol type="1"><li>The query string is empty (ie. its length is zero).</li>
<li><a href="#conf-docinfo" title="11.2.4.&nbsp;docinfo">docinfo</a> storage is set to <code class="code">extern</code>.</li>
</ol></div>
<p>
In full scan mode, all the indexed documents will be considered as matching.
Such queries will still apply filters, sorting, and group by, but will not perform any full-text searching.
This can be useful to unify full-text and non-full-text searching code, or to offload SQL server
(there are cases when Sphinx scans will perform better than analogous MySQL queries).
An example of using the full scan mode might be to find posts in a forum.
By selecting the forum's user ID via <code class="code">SetFilter()</code> but not actually providing any search text,
Sphinx will match every document (i.e. every post) where <code class="code">SetFilter()</code> would match -
in this case providing every post from that user. By default this will be ordered by relevancy,
followed by Sphinx document ID in ascending order (earliest first).
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="boolean-syntax"></a>5.2.&nbsp;Boolean query syntax</h2></div></div></div>
<p>
Boolean queries allow the following special operators to be used:
</p><div class="itemizedlist"><ul type="disc"><li>explicit operator AND: <pre class="programlisting">hello &amp; world</pre></li>
<li>operator OR: <pre class="programlisting">hello | world</pre></li>
<li>operator NOT:
<pre class="programlisting">
hello -world
hello !world
</pre></li>
<li>grouping: <pre class="programlisting">( hello world )</pre></li>
</ul></div>
<p>
Here's an example query which uses all these operators:
</p><div class="example"><a name="ex-boolean-query"></a><p class="title"><b>Example&nbsp;5.1.&nbsp;Boolean query example</b></p><div class="example-contents"><pre class="programlisting">
( cat -dog ) | ( cat -mouse)
</pre></div></div>
<p><br class="example-break">
</p><p>
There always is implicit AND operator, so "hello world" query actually
means "hello &amp; world".
</p><p>
OR operator precedence is higher than AND, so "looking for cat | dog | mouse"
means "looking for ( cat | dog | mouse )" and <span class="emphasis"><em>not</em></span>
"(looking for cat) | dog | mouse".
</p><p>
Queries like "-dog", which implicitly include all documents from the
collection, can not be evaluated. This is both for technical and performance
reasons. Technically, Sphinx does not always keep a list of all IDs.
Performance-wise, when the collection is huge (ie. 10-100M documents),
evaluating such queries could take very long.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="extended-syntax"></a>5.3.&nbsp;Extended query syntax</h2></div></div></div>
<p>
The following special operators and modifiers can be used when using the extended matching mode:
</p><div class="itemizedlist"><ul type="disc"><li>operator OR: <pre class="programlisting">hello | world</pre></li>
<li>operator NOT:
<pre class="programlisting">
hello -world
hello !world
</pre></li>
<li>field search operator: <pre class="programlisting">@title hello @body world</pre></li>
<li>field position limit modifier (introduced in version 0.9.9-rc1): <pre class="programlisting">@body[50] hello</pre></li>
<li>multiple-field search operator: <pre class="programlisting">@(title,body) hello world</pre></li>
<li>all-field search operator: <pre class="programlisting">@* hello</pre></li>
<li>phrase search operator: <pre class="programlisting">"hello world"</pre></li>
<li>proximity search operator: <pre class="programlisting">"hello world"~10</pre></li>
<li>quorum matching operator: <pre class="programlisting">"the world is a wonderful place"/3</pre></li>
<li>strict order operator (aka operator "before"): <pre class="programlisting">aaa &lt;&lt; bbb &lt;&lt; ccc</pre></li>
<li>exact form modifier (introduced in version 0.9.9-rc1): <pre class="programlisting">raining =cats and =dogs</pre></li>
<li>field-start and field-end modifier (introduced in version 0.9.9-rc2): <pre class="programlisting">^hello world$</pre></li>
</ul></div>
<p>

Here's an example query that uses some of these operators:
</p><div class="example"><a name="ex-extended-query"></a><p class="title"><b>Example&nbsp;5.2.&nbsp;Extended matching mode: query example</b></p><div class="example-contents"><pre class="programlisting">
"hello world" @title "example program"~5 @body python -(php|perl) @* code
</pre></div></div>
<p><br class="example-break">
The full meaning of this search is:

</p><div class="itemizedlist"><ul type="disc"><li>Find the words 'hello' and 'world' adjacently in any field in a document;</li>
<li>Additionally, the same document must also contain the words 'example' and 'program'
	in the title field, with up to, but not including, 10 words between the words in question;
	(E.g. "example PHP program" would be matched however "example script to introduce outside data
	into the correct context for your program" would not because two terms have 10 or more words between them)</li>
<li>Additionally, the same document must contain the word 'python' in the body field, but not contain either 'php' or 'perl';</li>
<li>Additionally, the same document must contain the word 'code' in any field.</li>
</ul></div>
<p>
</p><p>
There always is implicit AND operator, so "hello world" means that
both "hello" and "world" must be present in matching document.
</p><p>
OR operator precedence is higher than AND, so "looking for cat | dog | mouse"
means "looking for ( cat | dog | mouse )" and <span class="emphasis"><em>not</em></span>
"(looking for cat) | dog | mouse".
</p><p>
Field limit operator limits subsequent searching to a given field.
Normally, query will fail with an error message if given field name does not exist
in the searched index. However, that can be suppressed by specifying "@@relaxed"
option at the very beginning of the query:
</p><pre class="programlisting">
@@relaxed @nosuchfield my query
</pre><p>
This can be helpful when searching through heterogeneous indexes with
different schemas.
</p><p>
Field position limit, introduced in version 0.9.9-rc1, additionaly restricts the searching
to first N position within given field (or fields). For example, "@body[50] hello" will
<span class="bold"><strong>not</strong></span> match the documents where the keyword 'hello' occurs at position 51 and below
in the body.
</p><p>
Proximity distance is specified in words, adjusted for word count, and
applies to all words within quotes. For instance, "cat dog mouse"~5 query
means that there must be less than 8-word span which contains all 3 words,
ie. "CAT aaa bbb ccc DOG eee fff MOUSE" document will <span class="emphasis"><em>not</em></span>
match this query, because this span is exactly 8 words long.
</p><p>
Quorum matching operator introduces a kind of fuzzy matching.
It will only match those documents that pass a given threshold of given words.
The example above ("the world is a wonderful place"/3) will match all documents
that have at least 3 of the 6 specified words.
</p><p>
Strict order operator (aka operator "before"), introduced in version 0.9.9-rc2,
will match the document only if its argument keywords occur in the document
exactly in the query order. For instance, "black &lt;&lt; cat" query (without
quotes) will match the document "black and white cat" but <span class="emphasis"><em>not</em></span>
the "that cat was black" document. Order operator has the lowest priority.
It can be applied both to just keywords and more complex expressions,
ie. this is a valid query:
</p><pre class="programlisting">
(bag of words) &lt;&lt; "exact phrase" &lt;&lt; red|green|blue
</pre><p>
</p><p>
Exact form keyword modifier, introduced in version 0.9.9-rc1, will match the document only if the keyword occurred
in exactly the specified form. The default behaviour is to match the document
if the stemmed keyword matches. For instance, "runs" query will match both
the document that contains "runs" <span class="emphasis"><em>and</em></span> the document that
contains "running", because both forms stem to just "run" - while "=runs"
query will only match the first document. Exact form operator requires
<a href="#conf-index-exact-words" title="11.2.39.&nbsp;index_exact_words">index_exact_words</a> option to be enabled.
This is a modifier that affects the keyword and thus can be used within
operators such as phrase, proximity, and quorum operators.
</p><p>
Field-start and field-end keyword modifiers, introduced in version 0.9.9-rc2,
will make the keyword match only if it occurred at the very start or the very end
of a fulltext field, respectively. For instance, the query "^hello world$"
(with quotes and thus combining phrase operator and start/end modifiers)
will only match documents that contain at least one field that has exactly
these two keywords.
</p><p>
Starting with 0.9.9-rc1, arbitrarily nested brackets and negations are allowed.
However, the query must be possible to compute without involving an implicit
list of all documents:
</p><pre class="programlisting">
// correct query
aaa -(bbb -(ccc ddd))

// queries that are non-computable
-aaa
aaa | -bbb
</pre><p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="weighting"></a>5.4.&nbsp;Weighting</h2></div></div></div>
<p>
Specific weighting function (currently) depends on the search mode.
</p><p>
There are these major parts which are used in the weighting functions:
</p><div class="orderedlist"><ol type="1"><li>phrase rank,</li>
<li>statistical rank.</li>
</ol></div>
<p>
</p><p>
Phrase rank is based on a length of longest common subsequence
(LCS) of search words between document body and query phrase. So if
there's a perfect phrase match in some document then its phrase rank
would be the highest possible, and equal to query words count.
</p><p>
Statistical rank is based on classic BM25 function which only takes
word frequencies into account. If the word is rare in the whole database
(ie. low frequency over document collection) or mentioned a lot in specific
document (ie. high frequency over matching document), it receives more weight.
Final BM25 weight is a floating point number between 0 and 1.
</p><p>
In all modes, per-field weighted phrase ranks are computed as
a product of LCS multiplied by per-field weight speficifed by user.
Per-field weights are integer, default to 1, and can not be set
lower than 1.
</p><p>
In SPH_MATCH_BOOLEAN mode, no weighting is performed at all, every match weight
is set to 1.
</p><p>
In SPH_MATCH_ALL and SPH_MATCH_PHRASE modes, final weight is a sum of weighted phrase ranks.
</p><p>
In SPH_MATCH_ANY mode, the idea is essentially the same, but it also
adds a count of matching words in each field. Before that, weighted
phrase ranks are additionally mutliplied by a value big enough to
guarantee that higher phrase rank in <span class="bold"><strong>any</strong></span> field will make the
match ranked higher, even if it's field weight is low.
</p><p>
In SPH_MATCH_EXTENDED mode, final weight is a sum of weighted phrase
ranks and BM25 weight, multiplied by 1000 and rounded to integer.
</p><p>
This is going to be changed, so that MATCH_ALL and MATCH_ANY modes
use BM25 weights as well. This would improve search results in those
match spans where phrase ranks are equal; this is especially useful
for 1-word queries.
</p><p>
The key idea (in all modes, besides boolean) is that better subphrase
matches are ranked higher, and perfect matches are pulled to the top. Author's
experience is that this phrase proximity based ranking provides noticeably
better search quality than any statistical scheme alone (such as BM25,
which is commonly used in other search engines).
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sorting-modes"></a>5.5.&nbsp;Sorting modes</h2></div></div></div>
<p>
There are the following result sorting modes available:
</p><div class="itemizedlist"><ul type="disc"><li>SPH_SORT_RELEVANCE mode, that sorts by relevance in descending order (best matches first);</li>
<li>SPH_SORT_ATTR_DESC mode, that sorts by an attribute in descending order (bigger attribute values first);</li>
<li>SPH_SORT_ATTR_ASC mode, that sorts by an attribute in ascending order (smaller attribute values first);</li>
<li>SPH_SORT_TIME_SEGMENTS mode, that sorts by time segments (last hour/day/week/month) in descending order, and then by relevance in descending order;</li>
<li>SPH_SORT_EXTENDED mode, that sorts by SQL-like combination of columns in ASC/DESC order;</li>
<li>SPH_SORT_EXPR mode, that sorts by an arithmetic expression.</li>
</ul></div>
<p>
</p><p>
SPH_SORT_RELEVANCE ignores any additional parameters and always sorts matches
by relevance rank. All other modes require an additional sorting clause, with the
syntax depending on specific mode. SPH_SORT_ATTR_ASC, SPH_SORT_ATTR_DESC and
SPH_SORT_TIME_SEGMENTS modes require simply an attribute name.

SPH_SORT_RELEVANCE is equivalent to sorting by "@weight DESC, @id ASC" in extended sorting mode,
SPH_SORT_ATTR_ASC is equivalent to "attribute ASC, @weight DESC, @id ASC",
and SPH_SORT_ATTR_DESC to "attribute DESC, @weight DESC, @id ASC" respectively.
</p><h3>SPH_SORT_TIME_SEGMENTS mode</h3><p>
In SPH_SORT_TIME_SEGMENTS mode, attribute values are split into so-called
time segments, and then sorted by time segment first, and by relevance second.
</p><p>
The segments are calculated according to the <span class="emphasis"><em>current timestamp</em></span>
at the time when the search is performed, so the results would change over time.
The segments are as follows:
</p><div class="itemizedlist"><ul type="disc"><li>last hour,</li>
<li>last day,</li>
<li>last week,</li>
<li>last month,</li>
<li>last 3 months,</li>
<li>everything else.</li>
</ul></div>
<p>
These segments are hardcoded, but it is trivial to change them if necessary.
</p><p>
This mode was added to support searching through blogs, news headlines, etc.
When using time segments, recent records would be ranked higher because of segment,
but withing the same segment, more relevant records would be ranked higher -
unlike sorting by just the timestamp attribute, which would not take relevance
into account at all.
</p><h3><a name="sort-extended"></a>SPH_SORT_EXTENDED mode</h3><p>
In SPH_SORT_EXTENDED mode, you can specify an SQL-like sort expression
with up to 5 attributes (including internal attributes), eg:
</p><pre class="programlisting">
@relevance DESC, price ASC, @id DESC
</pre><p>
</p><p>
Both internal attributes (that are computed by the engine on the fly)
and user attributes that were configured for this index are allowed.
Internal attribute names must start with magic @-symbol; user attribute
names can be used as is. In the example above, <code class="option">@relevance</code>
and <code class="option">@id</code> are internal attributes and <code class="option">price</code> is user-specified.
</p><p>
Known internal attributes are:
</p><div class="itemizedlist"><ul type="disc"><li>@id (match ID)</li>
<li>@weight (match weight)</li>
<li>@rank (match weight)</li>
<li>@relevance (match weight)</li>
<li>@random (return results in random order)</li>
</ul></div>
<p>
<code class="option">@rank</code> and <code class="option">@relevance</code> are just additional
aliases to <code class="option">@weight</code>.
</p><h3><a name="sort-expr"></a>SPH_SORT_EXPR mode</h3><p>
Expression sorting mode lets you sort the matches by an arbitrary arithmetic
expression, involving attribute values, internal attributes (@id and @weight),
arithmetic operations, and a number of built-in functions. Here's an example:
</p><pre class="programlisting">
$cl-&gt;SetSortMode ( SPH_SORT_EXPR,
	"@weight + ( user_karma + ln(pageviews) )*0.1" );
</pre><p>
</p><p>
The following operators and functions are supported. They are mimiced after MySQL.
The functions take a number of arguments depending on the specific function.
</p><div class="itemizedlist"><ul type="disc"><li>Operators: +, -, *, /, &lt;, &gt; &lt;=, &gt;=, =, &lt;&gt;.</li>
<li>Boolean operators: AND, OR, NOT.</li>
<li>0-argument functions: NOW().</li>
<li>Unary (1-argument) functions: ABS(), CEIL(), FLOOR(), SIN(), COS(), LN(), LOG2(), LOG10(), EXP(), SQRT(), BIGINT(), SINT().</li>
<li>Binary (2-argument) functions: MIN(), MAX(), POW(), IDIV().</li>
<li>Other functions: IF(), INTERVAL(), IN(), GEODIST().</li>
</ul></div>
<p>
</p><p>
Calculations can be performed in three different modes: (a) using single-precision,
32-bit IEEE 754 floating point values (the default), (b) using signed 32-bit integers,
(c) using 64-bit signed integers. The expression parser will automatically switch
to integer mode if there are no operations the result in a floating point value.
Otherwise, it will use the default floating point mode. For instance, "a+b"
will be computed using 32-bit integers if both arguments are 32-bit integers;
or using 64-bit integers if both arguments are integers but one of them is
64-bit; or in floats otherwise. However, "a/b" or "sqrt(a)" will always be
computed in floats, because these operations return non-integer result.
To avoid the first, you can use IDIV(). Also, "a*b" will not be automatically
promoted to 64-bit when the arguments are 32-bit. To enforce 64-bit results,
you can use BIGINT(). (But note that if there are non-integer operations,
BIGINT() will simply be ignored.)
</p><p>
Comparison operators (eg. = or &lt;=) return 1.0 when the condition is true and 0.0 otherwise.
For instance, <code class="code">(a=b)+3</code> will evaluate to 4 when attribute 'a' is equal to attribute 'b', and to 3 when 'a' is not.
Unlike MySQL, the equality comparisons (ie. = and &lt;&gt; operators) introduce a small equality threshold (1e-6 by default).
If the difference between compared values is within the threshold, they will be considered equal.
</p><p>
Boolean operators (AND, OR, NOT) were introduced in 0.9.9-rc2 and behave as usual.
They are left-associative and have the least priority compared to other operators.
NOT has more priority than AND and OR but nevertheless less than any other operator.
AND and OR have the same priority so brackets use is recommended to avoid confusion
in complex expressions.
</p><p>
All unary and binary functions are straightforward, they behave just like their mathematical counterparts.
But <code class="code">IF()</code> behavior needs to be explained in more detail.
It takes 3 arguments, check whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero, or the 3rd one when it is.
Note that unlike comparison operators, <code class="code">IF()</code> does <span class="bold"><strong>not</strong></span> use a threshold!
Therefore, it's safe to use comparison results as its 1st argument, but arithmetic operators might produce unexpected results.
For instance, the following two calls will produce <span class="emphasis"><em>different</em></span> results even though they are logically equivalent:
</p><pre class="programlisting">
IF ( sqrt(3)*sqrt(3)-3&lt;&gt;0, a, b )
IF ( sqrt(3)*sqrt(3)-3, a, b )
</pre><p>
In the first case, the comparison operator &lt;&gt; will return 0.0 (false)
because of a threshold, and <code class="code">IF()</code> will always return 'b' as a result.
In the second one, the same <code class="code">sqrt(3)*sqrt(3)-3</code> expression will be compared
with zero <span class="emphasis"><em>without</em></span> threshold by the <code class="code">IF()</code> function itself.
But its value will be slightly different from zero because of limited floating point
calculations precision. Because of that, the comparison with 0.0 done by <code class="code">IF()</code>
will not pass, and the second variant will return 'a' as a result.
</p><p>
BIGINT() function, introduced in version 0.9.9-rc1, forcibly promotes the integer argument to 64-bit type,
and does nothing on floating point argument. It's intended to help enforce evaluation
of certain expressions (such as "a*b") in 64-bit mode even though all the arguments
are 32-bit.
</p><p>
SINT() function, introduced in version 1.10-beta, forcibly reinterprets its
32-bit unsigned integer argument as signed, and also expands it to 64-bit type
(because 32-bit type is unsigned). It's easily illustrated by the following
example: 1-2 normally evaluates to 4294967295, but SINT(1-2) evaluates to -1.
</p><p>
IDIV() functions performs an integer division on its 2 arguments. The result
is integer as well, unlike "a/b" result.
</p><p>
IN(expr,val1,val2,...), introduced in version 0.9.9-rc1, takes 2 or more arguments, and returns 1 if 1st argument
(expr) is equal to any of the other arguments (val1..valN), or 0 otherwise.
Currently, all the checked values (but not the expression itself!) are required
to be constant. (Its technically possible to implement arbitrary expressions too,
and that might be implemented in the future.) Constants are pre-sorted and then
binary search is used, so IN() even against a big arbitrary list of constants
will be very quick. Starting with 0.9.9-rc2, first argument can also be
a MVA attribute. In that case, IN() will return 1 if any of the MVA values
is equal to any of the other arguments.
</p><p>
INTERVAL(expr,point1,point2,point3,...), introduced in version 0.9.9-rc1, takes 2 or more arguments, and returns
the index of the argument that is less than the first argument: it returns
0 if expr&lt;point1, 1 if point1&lt;=expr&lt;point2, and so on.
It is required that point1&lt;point2&lt;...&lt;pointN for this function
to work correctly.
</p><p>
NOW(), introduced in version 0.9.9-rc1, is a helper function that returns current timestamp as a 32-bit integer.
</p><p>
GEODIST(lat1,long1,lat2,long2) function, introduced in version 0.9.9-rc2,
computes geosphere distance between two given points specified by their
coordinates. Note that both latitudes and longitudes must be in radians
and the result will be in meters. You can use arbitrary expression as any
of the four coordinates. An optimized path will be selected when one pair
of the arguments refers directly to a pair attributes and the other one
is constant.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="clustering"></a>5.6.&nbsp;Grouping (clustering) search results </h2></div></div></div>
<p>
Sometimes it could be useful to group (or in other terms, cluster)
search results and/or count per-group match counts - for instance,
to draw a nice graph of how much maching blog posts were there per
each month; or to group Web search results by site; or to group
matching forum posts by author; etc.
</p><p>
In theory, this could be performed by doing only the full-text search
in Sphinx and then using found IDs to group on SQL server side. However,
in practice doing this with a big result set (10K-10M matches) would
typically kill performance.
</p><p>
To avoid that, Sphinx offers so-called grouping mode. It is enabled
with SetGroupBy() API call. When grouping, all matches are assigned to
different groups based on group-by value. This value is computed from
specified attribute using one of the following built-in functions:
</p><div class="itemizedlist"><ul type="disc"><li>SPH_GROUPBY_DAY, extracts year, month and day in YYYYMMDD format from timestamp;</li>
<li>SPH_GROUPBY_WEEK, extracts year and first day of the week number (counting from year start) in YYYYNNN format from timestamp;</li>
<li>SPH_GROUPBY_MONTH, extracts month in YYYYMM format from timestamp;</li>
<li>SPH_GROUPBY_YEAR, extracts year in YYYY format from timestamp;</li>
<li>SPH_GROUPBY_ATTR, uses attribute value itself for grouping.</li>
</ul></div>
<p>
</p><p>
The final search result set then contains one best match per group.
Grouping function value and per-group match count are returned along
as "virtual" attributes named
<span class="bold"><strong>@group</strong></span> and
<span class="bold"><strong>@count</strong></span> respectively.
</p><p>
The result set is sorted by group-by sorting clause, with the syntax similar
to <a href="#sort-extended"><code class="option">SPH_SORT_EXTENDED</code> sorting clause</a>
syntax. In addition to <code class="option">@id</code> and <code class="option">@weight</code>,
group-by sorting clause may also include:
</p><div class="itemizedlist"><ul type="disc"><li>@group (groupby function value),</li>
<li>@count (amount of matches in group).</li>
</ul></div>
<p>
</p><p>
The default mode is to sort by groupby value in descending order,
ie. by <code class="option">"@group desc"</code>.
</p><p>
On completion, <code class="option">total_found</code> result parameter would
contain total amount of matching groups over he whole index.
</p><p>
<span class="bold"><strong>WARNING:</strong></span> grouping is done in fixed memory
and thus its results are only approximate; so there might be more groups reported
in <code class="option">total_found</code> than actually present. <code class="option">@count</code> might also
be underestimated. To reduce inaccuracy, one should raise <code class="option">max_matches</code>.
If <code class="option">max_matches</code> allows to store all found groups, results will be 100% correct.
</p><p>
For example, if sorting by relevance and grouping by <code class="code">"published"</code>
attribute with <code class="code">SPH_GROUPBY_DAY</code> function, then the result set will
contain
</p><div class="itemizedlist"><ul type="disc"><li>one most relevant match per each day when there were any
matches published,</li>
<li>with day number and per-day match count attached,</li>
<li>sorted by day number in descending order (ie. recent days first).</li>
</ul></div>
<p>
</p><p>
Starting with version 0.9.9-rc2, aggregate functions (AVG(), MIN(),
MAX(), SUM()) are supported through <a href="#api-func-setselect" title="8.2.4.&nbsp;SetSelect">SetSelect()</a> API call
when using GROUP BY.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="distributed"></a>5.7.&nbsp;Distributed searching</h2></div></div></div>
<p>
To scale well, Sphinx has distributed searching capabilities.
Distributed searching is useful to improve query latency (ie. search
time) and throughput (ie. max queries/sec) in multi-server, multi-CPU
or multi-core environments. This is essential for applications which
need to search through huge amounts data (ie. billions of records
and terabytes of text).
</p><p>
The key idea is to horizontally partition (HP) searched data
accross search nodes and then process it in parallel.
</p><p>
Partitioning is done manually. You should
</p><div class="itemizedlist"><ul type="disc"><li>setup several instances
of Sphinx programs (<code class="filename">indexer</code> and <code class="filename">searchd</code>)
on different servers;</li>
<li>make the instances index (and search) different parts of data;</li>
<li>configure a special distributed index on some of the <code class="filename">searchd</code>
instances;</li>
<li>and query this index.</li>
</ul></div>
<p>
This index only contains references to other
local and remote indexes - so it could not be directly reindexed,
and you should reindex those indexes which it references instead.
</p><p>
When <code class="filename">searchd</code> receives a query against distributed index,
it does the following:
</p><div class="orderedlist"><ol type="1"><li>connects to configured remote agents;</li>
<li>issues the query;</li>
<li>sequentially searches configured local indexes (while the remote agents are searching);</li>
<li>retrieves remote agents' search results;</li>
<li>merges all the results together, removing the duplicates;</li>
<li>sends the merged resuls to client.</li>
</ol></div>
<p>
</p><p>
From the application's point of view, there are no differences
between searching through a regular index, or a distributed index at all.
That is, distributed indexes are fully transparent to the application,
and actually there's no way to tell whether the index you queried
was distributed or local. (Even though as of 0.9.9 Sphinx does not
allow to combine searching through distributed indexes with anything else,
this constraint will be lifted in the future.)
</p><p>
Any <code class="filename">searchd</code> instance could serve both as a master
(which aggregates the results) and a slave (which only does local searching)
at the same time. This has a number of uses:
</p><div class="orderedlist"><ol type="1"><li>every machine in a cluster could serve as a master which
searches the whole cluster, and search requests could be balanced between
masters to achieve a kind of HA (high availability) in case any of the nodes fails;
</li>
<li>
if running within a single multi-CPU or multi-core machine, there
would be only 1 searchd instance quering itself as an agent and thus
utilizing all CPUs/core.
</li>
</ol></div>
<p>
</p><p>
It is scheduled to implement better HA support which would allow
to specify which agents mirror each other, do health checks, keep track
of alive agents, load-balance requests, etc.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="query-log-format"></a>5.8.&nbsp;<code class="filename">searchd</code> query log format</h2></div></div></div>
<p>
<code class="filename">searchd</code> logs all succesfully executed search queries
into query log file. Here's an example:

</p><pre class="programlisting">
[Fri Jun 29 21:17:58 2007] 0.004 sec [all/0/rel 35254 (0,20)] [lj] test
[Fri Jun 29 21:20:34 2007] 0.024 sec [all/0/rel 19886 (0,20) @channel_id] [lj] test
</pre><p>

This log format is as follows:
</p><pre class="programlisting">
[query-date] query-time [match-mode/filters-count/sort-mode
    total-matches (offset,limit) @groupby-attr] [index-name] query
</pre><p>

Match mode can take one of the following values:
</p><div class="itemizedlist"><ul type="disc"><li>"all" for SPH_MATCH_ALL mode;</li>
<li>"any" for SPH_MATCH_ANY mode;</li>
<li>"phr" for SPH_MATCH_PHRASE mode;</li>
<li>"bool" for SPH_MATCH_BOOLEAN mode;</li>
<li>"ext" for SPH_MATCH_EXTENDED mode;</li>
<li>"ext2" for SPH_MATCH_EXTENDED2 mode;</li>
<li>"scan" if the full scan mode was used, either by being specified with SPH_MATCH_FULLSCAN, or if the query was empty (as documented under <a href="#matching-modes" title="5.1.&nbsp;Matching modes">Matching Modes</a>)</li>
</ul></div>
<p>

Sort mode can take one of the following values:

</p><div class="itemizedlist"><ul type="disc"><li>"rel" for SPH_SORT_RELEVANCE mode;</li>
<li>"attr-" for SPH_SORT_ATTR_DESC mode;</li>
<li>"attr+" for SPH_SORT_ATTR_ASC mode;</li>
<li>"tsegs" for SPH_SORT_TIME_SEGMENTS mode;</li>
<li>"ext" for SPH_SORT_EXTENDED mode.</li>
</ul></div>
<p>

</p><p>Additionally, if <code class="filename">searchd</code> was started with <code class="option">--iostats</code>, there will be a block of data after where the index(es) searched are listed.</p><p>A query log entry might take the form of:</p><pre class="programlisting">
[Fri Jun 29 21:17:58 2007] 0.004 sec [all/0/rel 35254 (0,20)] [lj]
   [ios=6 kb=111.1 ms=0.5] test
</pre><p>
This additional block is information regarding I/O operations in performing the search:
the number of file I/O operations carried out, the amount of data in kilobytes read from
the index files and time spent on I/O operations (although there is a background processing
component, the bulk of this time is the I/O operation time).
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql"></a>5.9.&nbsp;MySQL protocol support and SphinxQL</h2></div></div></div>
<p>
Starting with version 0.9.9-rc2, Sphinx searchd daemon supports MySQL binary
network protocol and can be accessed with regular MySQL API. For instance,
'mysql' CLI client program works well. Here's an example of querying
Sphinx using MySQL client:
</p><pre class="programlisting">
$ mysql -P 9306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 0.9.9-dev (r1734)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql&gt; SELECT * FROM test1 WHERE MATCH('test') 
    -&gt; ORDER BY group_id ASC OPTION ranker=bm25;
+------+--------+----------+------------+
| id   | weight | group_id | date_added |
+------+--------+----------+------------+
|    4 |   1442 |        2 | 1231721236 |
|    2 |   2421 |      123 | 1231721236 |
|    1 |   2421 |      456 | 1231721236 |
+------+--------+----------+------------+
3 rows in set (0.00 sec)
</pre><p>
</p><p>
Note that mysqld was not even running on the test machine. Everything was
handled by searchd itself.
</p><p>
The new access method is supported <span class="emphasis"><em>in addition</em></span>
to native APIs which all still work perfectly well. In fact, both
access methods can be used at the same time. Also, native API is still
the default access method. MySQL protocol support needs to be additionally
configured. This is a matter of 1-line config change, adding a new
<a href="#conf-listen" title="11.4.1.&nbsp;listen">listener</a> with mysql41 specified
as a protocol:
</p><pre class="programlisting">
listen = localhost:9306:mysql41
</pre><p>
</p><p>
Just supporting the protocol and not the SQL syntax would be useless
so Sphinx now also supports a subset of SQL that we dubbed SphinxQL.
It supports the standard querying all the index types with SELECT,
modifying RT indexes with INSERT, REPLACE, and DELETE, and much more.
Full SphinxQL reference is available in <a href="#sphinxql-reference" title="Chapter&nbsp;7.&nbsp;SphinxQL reference">Chapter&nbsp;7, <i>SphinxQL reference</i></a>.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="multi-queries"></a>5.10.&nbsp;Multi-queries</h2></div></div></div>
<p>
Multi-queries, or query batches, let you send multiple queries to Sphinx
in one go (more formally, one network request).
</p><p>
Two API methods that implement multi-query mechanism are
<a href="#api-func-addquery" title="8.6.2.&nbsp;AddQuery">AddQuery()</a> and
<a href="#api-func-runqueries" title="8.6.3.&nbsp;RunQueries">RunQueries()</a>.
(In fact, regular <a href="#api-func-addquery" title="8.6.2.&nbsp;AddQuery">Query()</a>
call is internally implemented as a single AddQuery() call immediately
followed by RunQueries() call.) AddQuery() captures the current state
of all the query settings set by previous API calls, and memorizes
the query. RunQueries() actually sends all the memorized queries,
and returns multiple result sets. There are no restrictions on
the queries at all, except just a sanity check on a number of queries
in a single batch (see <a href="#conf-max-batch-queries" title="11.4.24.&nbsp;max_batch_queries">Section&nbsp;11.4.24, &#8220;max_batch_queries&#8221;</a>).
</p><p>
Why use multi-queries? Generally, it all boils down to performance.
First, by sending requests to <code class="filename">searchd</code> in a batch
instead of one by one, you always save a bit by doing less network
roundtrips. Second, and somewhat more important, sending queries
in a batch enables <code class="filename">searchd</code> to perform certain
internal optimizations. As new types of optimizations are being
added over time, it generally makes sense to pack all the queries
into batches where possible, so that simply upgrading Sphinx
to a new version would automatically enable new optimizations.
In the case when there aren't any possible batch optimizations
to apply, queries will be processed one by one internally.
</p><p>
Why (or rather when) not use multi-queries? Multi-queries requires
all the queries in a batch to be independent, and sometimes they aren't.
That is, sometimes query B is based on query A results, and so can only be
set up after executing query A. For instance, you might want to display
results from a secondary index if and only if there were no results
found in a primary index. Or maybe just specify offset into 2nd result set
based on the amount of matches in the 1st result set. In that case,
you will have to use separate queries (or separate batches).
</p><p>
As of 0.9.10, there are two major optimizations to be aware of:
common query optimization (available since 0.9.8); and common
subtree optimization (available since 0.9.10).
</p><p>
<span class="bold"><strong>Common query optimization</strong></span> means that <code class="filename">searchd</code>
will identify all those queries in a batch where only the sorting
and group-by settings differ, and <span class="emphasis"><em>only perform searching once</em></span>.
For instance, if a batch consists of 3 queries, all of them are for
"ipod nano", but 1st query requests top-10 results sorted by price,
2nd query groups by vendor ID and requests top-5 vendors sorted by
rating, and 3rd query requests max price, full-text search for
"ipod nano" will only be performed once, and its results will be
reused to build 3 different result sets.
</p><p>
So-called <span class="bold"><strong>faceted searching</strong></span> is a particularly important case
that benefits from this optimization. Indeed, faceted searching
can be implemented by running a number of queries, one to retrieve
search results themselves, and a few other ones with same full-text
query but different  group-by settings to retrieve all the required
groups of results (top-3 authors, top-5 vendors, etc). And as long
as full-text query and filtering settings stay the same, common
query optimization will trigger, and greatly improve performance.
</p><p>
<span class="bold"><strong>Common subtree optimization</strong></span> is even more interesting.
It lets <code class="filename">searchd</code> exploit similarities between
batched full-text queries. It identifies common full-text query parts
(subtress) in all queries, and caches them between queries. For instance,
look at the following query batch:
</p><pre class="programlisting">
barack obama president
barack obama john mccain
barack obama speech
</pre><p>
There's a common two-word part ("barack obama") that can be computed
only once, then cached and shared across the queries. And common subtree
optimization does just that. Per-query cache size is strictly controlled
by <a href="#conf-subtree-docs-cache" title="11.4.25.&nbsp;subtree_docs_cache">subtree_docs_cache</a>
and <a href="#conf-subtree-hits-cache" title="11.4.26.&nbsp;subtree_hits_cache">subtree_hits_cache</a>
directives (so that caching <span class="emphasis"><em>all</em></span> sxiteen gazillions
of documents that match "i am" does not exhaust the RAM and instantly
kill your server).
</p><p>
Here's a code sample (in PHP) that fire the same query in 3 different
sorting modes:
</p><pre class="programlisting">
require ( "sphinxapi.php" );
$cl = new SphinxClient ();
$cl-&gt;SetMatchMode ( SPH_MATCH_EXTENDED2 );

$cl-&gt;SetSortMode ( SPH_SORT_RELEVANCE );
$cl-&gt;AddQuery ( "the", "lj" );
$cl-&gt;SetSortMode ( SPH_SORT_EXTENDED, "published desc" );
$cl-&gt;AddQuery ( "the", "lj" );
$cl-&gt;SetSortMode ( SPH_SORT_EXTENDED, "published asc" );
$cl-&gt;AddQuery ( "the", "lj" );
$res = $cl-&gt;RunQueries();
</pre><p>
</p><p>
How to tell whether the queries in the batch were actually optimized?
If they were, respective query log will have a "multiplier" field that
specifies how many queries were processed together:
</p><pre class="programlisting">
[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext2/0/rel 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext2/0/ext 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext2/0/ext 747541 (0,20)] [lj] the
</pre><p>
Note the "x3" field. It means that this query was optimized and
processed in a sub-batch of 3 queries. For reference, this is how
the regular log would look like if the queries were not batched:
</p><pre class="programlisting">
[Sun Jul 12 15:18:17.062 2009] 0.059 sec [ext2/0/rel 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.156 2009] 0.091 sec [ext2/0/ext 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.250 2009] 0.092 sec [ext2/0/ext 747541 (0,20)] [lj] the
</pre><p>
Note how per-query time in multi-query case was improved by a factor 
of 1.5x to 2.3x, depending on a particular sorting mode. In fact, for both
common query and common subtree optimizations, there were reports of 3x and
even more improvements, and that's from production instances, not just
synthetic tests.
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="command-line-tools"></a>Chapter&nbsp;6.&nbsp;Command line tools reference</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#ref-indexer">6.1. <code class="filename">indexer</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-searchd">6.2. <code class="filename">searchd</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-search">6.3. <code class="filename">search</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-spelldump">6.4. <code class="filename">spelldump</code> command reference</a></span></dt>
<dt><span class="sect1"><a href="#ref-indextool">6.5. <code class="filename">indextool</code> command reference</a></span></dt>
</dl></div>
<p>As mentioned elsewhere, Sphinx is not a single program called 'sphinx',
but a collection of 4 separate programs which collectively form Sphinx. This section
covers these tools and how to use them.</p><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ref-indexer"></a>6.1.&nbsp;<code class="filename">indexer</code> command reference</h2></div></div></div>
<p><code class="filename">indexer</code> is the first of the two principle tools
as part of Sphinx. Invoked from either the command line directly, or as part
of a larger script, <code class="filename">indexer</code> is solely responsible
for gathering the data that will be searchable.</p><p>The calling syntax for <code class="filename">indexer</code> is as follows:</p><pre class="programlisting">
indexer [OPTIONS] [indexname1 [indexname2 [...]]]
</pre><p>Essentially you would list the different possible indexes (that you would later
make available to search) in <code class="filename">sphinx.conf</code>, so when calling
<code class="filename">indexer</code>, as a minimum you need to be telling it what index
(or indexes) you want to index.</p><p>If <code class="filename">sphinx.conf</code> contained details on 2 indexes,
<code class="filename">mybigindex</code> and <code class="filename">mysmallindex</code>,
you could do the following:</p><pre class="programlisting">
$ indexer mybigindex
$ indexer mysmallindex mybigindex
</pre><p>As part of the configuration file, <code class="filename">sphinx.conf</code>, you specify
one or more indexes for your data. You might call <code class="filename">indexer</code> to reindex
one of them, ad-hoc, or you can tell it to process all indexes - you are not limited
to calling just one, or all at once, you can always pick some combination
of the available indexes.</p><p>The majority of the options for <code class="filename">indexer</code> are given
in the configuration file, however there are some options you might need to specify
on the command line as well, as they can affect how the indexing operation is performed.
These options are:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--config &lt;file&gt;</code> (<code class="option">-c &lt;file&gt;</code> for short)
tells <code class="filename">indexer</code> to use the given file as its configuration. Normally,
it will look for <code class="filename">sphinx.conf</code> in the installation directory
(e.g. <code class="filename">/usr/local/sphinx/etc/sphinx.conf</code> if installed into
<code class="filename">/usr/local/sphinx</code>), followed by the current directory you are
in when calling <code class="filename">indexer</code> from the shell. This is most of use
in shared environments where the binary files are installed somewhere like
<code class="filename">/usr/local/sphinx/</code> but you want to provide users with
the ability to make their own custom Sphinx set-ups, or if you want to run
multiple instances on a single server. In cases like those you could allow them
to create their own <code class="filename">sphinx.conf</code> files and pass them to
<code class="filename">indexer</code> with this option. For example:
<pre class="programlisting">
$ indexer --config /home/myuser/sphinx.conf myindex
</pre></li>
<li><code class="option">--all</code> tells <code class="filename">indexer</code> to update
every index listed in <code class="filename">sphinx.conf</code>, instead of listing individual indexes.
This would be useful in small configurations, or <code class="filename">cron</code>-type or maintenance
jobs where the entire index set will get rebuilt each day, or week, or whatever period is best.
Example usage:
<pre class="programlisting">
$ indexer --config /home/myuser/sphinx.conf --all
</pre></li>
<li><code class="option">--rotate</code> is used for rotating indexes. Unless you have the situation
where you can take the search function offline without troubling users, you will almost certainly
need to keep search running whilst indexing new documents. <code class="option">--rotate</code> creates
a second index, parallel to the first (in the same place, simply including <code class="filename">.new</code>
in the filenames). Once complete, <code class="filename">indexer</code> notifies <code class="filename">searchd</code>
via sending the <code class="option">SIGHUP</code> signal, and <code class="filename">searchd</code> will attempt
to rename the indexes (renaming the existing ones to include <code class="filename">.old</code>
and renaming the <code class="filename">.new</code> to replace them), and then start serving
from the newer files. Depending on the setting of
<a href="#conf-seamless-rotate" title="11.4.11.&nbsp;seamless_rotate">seamless_rotate</a>, there may be a slight delay
in being able to search the newer indexes. Example usage:
<pre class="programlisting">
$ indexer --rotate --all
</pre></li>
<li><code class="option">--quiet</code> tells <code class="filename">indexer</code> not to output anything,
unless there is an error. Again, most used for <code class="filename">cron</code>-type, or other script
jobs where the output is irrelevant or unnecessary, except in the event of some kind of error.
Example usage:
<pre class="programlisting">
$ indexer --rotate --all --quiet
</pre></li>
<li><code class="option">--noprogress</code> does not display progress details as they occur;
instead, the final status details (such as documents indexed, speed of indexing and so on
are only reported at completion of indexing. In instances where the script is not being
run on a console (or 'tty'), this will be on by default. Example usage:
<pre class="programlisting">
$ indexer --rotate --all --noprogress
</pre></li>
<li><code class="option">--buildstops &lt;outputfile.text&gt; &lt;N&gt;</code> reviews
the index source, as if it were indexing the data, and produces a list of the terms
that are being indexed. In other words, it produces a list of all the searchable terms
that are becoming part of the index. Note; it does not update the index in question,
it simply processes the data 'as if' it were indexing, including running queries
defined with <code class="option">sql_query_pre</code> or <code class="option">sql_query_post</code>.
<code class="filename">outputfile.txt</code> will contain the list of words, one per line,
sorted by frequency with most frequent first, and <code class="filename">N</code> specifies
the maximum number of words that will be listed; if sufficiently large to encompass
every word in the index, only that many words will be returned. Such a dictionary list
could be used for client application features around "Did you mean..." functionality,
usually in conjunction with <code class="option">--buildfreqs</code>, below. Example:
<pre class="programlisting">
$ indexer myindex --buildstops word_freq.txt 1000
</pre>
This would produce a document in the current directory, <code class="filename">word_freq.txt</code>
with the 1,000 most common words in 'myindex', ordered by most common first. Note that
the file will pertain to the last index indexed when specified with multiple indexes or
<code class="option">--all</code> (i.e. the last one listed in the configuration file)
</li>
<li><code class="option">--buildfreqs</code> works with <code class="option">--buildstops</code>
(and is ignored if <code class="option">--buildstops</code> is not specified).
As <code class="option">--buildstops</code> provides the list of words used within the index,
<code class="option">--buildfreqs</code> adds the quantity present in the index, which would be
useful in establishing whether certain words should be considered stopwords
if they are too prevalent. It will also help with developing "Did you mean..."
features where you can how much more common a given word compared to another,
similar one. Example:
<pre class="programlisting">
$ indexer myindex --buildstops word_freq.txt 1000 --buildfreqs
</pre>
This would produce the <code class="filename">word_freq.txt</code> as above, however after each word would be the number of times it occurred in the index in question.
</li>
<li><code class="option">--merge &lt;dst-index&gt; &lt;src-index&gt;</code> is used
for physically merging indexes together, for example if you have a main+delta scheme,
where the main index rarely changes, but the delta index is rebuilt frequently,
and <code class="option">--merge</code> would be used to combine the two. The operation moves
from right to left - the contents of <code class="filename">src-index</code> get examined
and physically combined with the contents of <code class="filename">dst-index</code>
and the result is left in <code class="filename">dst-index</code>.
In pseudo-code, it might be expressed as: <code class="code">dst-index += src-index</code>
An example:
<pre class="programlisting">
$ indexer --merge main delta --rotate
</pre>
In the above example, where the main is the master, rarely modified index,
and delta is the less frequently modified one, you might use the above to call
<code class="filename">indexer</code> to combine the contents of the delta into the
main index and rotate the indexes.
</li>
<li><code class="option">--merge-dst-range &lt;attr&gt; &lt;min&gt; &lt;max&gt;</code>
runs the filter range given upon merging. Specifically, as the merge is applied
to the destination index (as part of <code class="option">--merge</code>, and is ignored 
if <code class="option">--merge</code> is not specified), <code class="filename">indexer</code>
will also filter the documents ending up in the destination index, and only
documents will pass through the filter given will end up in the final index.
This could be used for example, in an index where there is a 'deleted' attribute,
where 0 means 'not deleted'. Such an index could be merged with:
<pre class="programlisting">
$ indexer --merge main delta --merge-dst-range deleted 0 0
</pre>
Any documents marked as deleted (value 1) would be removed from the newly-merged
destination index. It can be added several times to the command line,
to add successive filters to the merge, all of which must be met in order
for a document to become part of the final index.
</li>
<li><code class="option">--dump-rows &lt;FILE&gt;</code> dumps rows fetched
by SQL source(s) into the specified file, in a MySQL compatible syntax.
Resulting dumps are the exact representation of data as received by
<code class="filename">indexer</code> and help to repeat indexing-time issues.
</li>
<li><code class="option">--verbose</code> guarantees that every row that
caused problems indexing (duplicate, zero, or missing document ID;
or file field IO issues; etc) will be reported. By default, this option
is off, and problem summaries may be reported instead.
</li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ref-searchd"></a>6.2.&nbsp;<code class="filename">searchd</code> command reference</h2></div></div></div>
<p><code class="filename">searchd</code> is the second of the two principle tools as part of Sphinx.
<code class="filename">searchd</code> is the part of the system which actually handles searches;
it functions as a server and is responsible for receiving queries, processing them and
returning a dataset back to the different APIs for client applications.</p><p>Unlike <code class="filename">indexer</code>, <code class="filename">searchd</code> is not designed
to be run either from a regular script or command-line calling, but instead either
as a daemon to be called from init.d (on Unix/Linux type systems) or to be called
as a service (on Windows-type systems), so not all of the command line options will
always apply, and so will be build-dependent.</p><p>Calling <code class="filename">searchd</code> is simply a case of:</p><pre class="programlisting">
$ searchd [OPTIONS]
</pre><p>The options available to <code class="filename">searchd</code> on all builds are:</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--help</code> (<code class="option">-h</code> for short) lists all of the
	parameters that can be called in your particular build of <code class="filename">searchd</code>.
</li>
<li><code class="option">--config &lt;file&gt;</code> (<code class="option">-c &lt;file&gt;</code> for short)
	tells <code class="filename">searchd</code> to use the given file as its configuration,
	just as with <code class="filename">indexer</code> above.
</li>
<li><code class="option">--stop</code> is used to asynchronously stop <code class="filename">searchd</code>,
	using the details of the PID file as specified in the <code class="filename">sphinx.conf</code> file,
	so you may also need to confirm to <code class="filename">searchd</code> which configuration
	file to use with the <code class="option">--config</code> option. NB, calling <code class="option">--stop</code>
	will also make sure any changes applied to the indexes with
	<a href="#api-func-updateatttributes" title="8.7.2.&nbsp;UpdateAttributes"><code class="code">UpdateAttributes()</code></a>
	will be applied to the index files themselves. Example:
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --stop
</pre></li>
<li><code class="option">--stopwait</code> is used to synchronously stop <code class="filename">searchd</code>.
<code class="option">--stop</code> essentially tells the running instance to exit (by sending it a SIGTERM)
and then immediately returns. <code class="option">--stopwait</code> will also attempt to wait until the
running <code class="filename">searchd</code> instance actually finishes the shutdown (eg. saves all
the pending attribute changes) and exits. Example:
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --stopwait
</pre>
Possible exit codes are as follows:
	<div class="itemizedlist"><ul type="circle"><li>0 on success;</li>
<li>1 if connection to running searchd daemon failed;</li>
<li>2 if daemon reported an error during shutdown;</li>
<li>3 if daemon crashed during shutdown.</li>
</ul></div></li>
<li><code class="option">--status</code> command is used to query running
<code class="filename">searchd</code> instance status, using the connection details
from the (optionally) provided configuration file. It will try to connect
to the running instance using the first configured UNIX socket or TCP port.
On success, it will query for a number of status and performance counter
values and print them. You can use <a href="#api-func-status" title="8.7.5.&nbsp;Status">Status()</a>
API call to access the very same counters from your application. Examples:
<pre class="programlisting">
$ searchd --status
$ searchd --config /home/myuser/sphinx.conf --status
</pre></li>
<li><code class="option">--pidfile</code> is used to explicitly state a PID file,
where the process information is stored regarding <code class="filename">searchd</code>,
used for inter-process communications (for example, <code class="filename">indexer</code>
will need to know the PID to contact <code class="filename">searchd</code> for rotating
indexes). Normally, <code class="filename">searchd</code> would use a PID if running
in regular mode (i.e. not with <code class="option">--console</code>), but it is possible
that you will be running it in console mode whilst the index is being updated
and rotated, for which a PID file will be needed.
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --pidfile /home/myuser/sphinx.pid
</pre></li>
<li><code class="option">--console</code> is used to force <code class="filename">searchd</code>
into console mode; typically it will be running as a conventional server application,
and will aim to dump information into the log files (as specified in
<code class="filename">sphinx.conf</code>). Sometimes though, when debugging issues
in the configuration or the daemon itself, or trying to diagnose hard-to-track-down
problems, it may be easier to force it to dump information directly
to the console/command line from which it is being called. Running in console mode
also means that the process will not be forked (so searches are done in sequence)
and logs will not be written to. (It should be noted that console mode
is not the intended method for running <code class="filename">searchd</code>.)
You can invoke it as such:
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --console
</pre></li>
<li><code class="option">--logdebug</code> enables additional debug output
in the daemon log. Should only be needed rarely, to assist with debugging
issues that could not be easily reproduced on request.
</li>
<li><code class="option">--iostats</code> is used in conjuction with the
logging options (the <code class="option">query_log</code> will need to have been
activated in <code class="filename">sphinx.conf</code>) to provide more detailed
information on a per-query basis as to the input/output operations
carried out in the course of that query, with a slight performance hit
and of course bigger logs. Further details are available under the
<a href="#query-log-format" title="5.8.&nbsp;searchd query log format">query log format</a> section.
You might start <code class="filename">searchd</code> thus:
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --iostats
</pre></li>
<li><code class="option">--cpustats</code> is used to provide actual CPU time
report (in addition to wall time) in both query log file (for every given
query) and status report (aggregated). It depends on clock_gettime() system
call and might therefore be unavailable on certain systems. You might start
<code class="filename">searchd</code> thus:
<pre class="programlisting">
$ searchd --config /home/myuser/sphinx.conf --cpustats
</pre></li>
<li><code class="option">--port portnumber</code> (<code class="option">-p</code> for short)
is used to specify the post that <code class="filename">searchd</code> should listen on,
usually for debugging purposes. This will usually default to 9312, but sometimes
you need to run it on a different port. Specifying it on the command line
will override anything specified in the configuration file. The valid range
is 0 to 65535, but ports numbered 1024 and below usually require
a privileged account in order to run. An example of usage:
<pre class="programlisting">
$ searchd --port 9313
</pre></li>
<li><code class="option">--index &lt;index&gt;</code> forces this instance
of <code class="filename">searchd</code> only to serve the specified index.
Like <code class="option">--port</code>, above, this is usually for debugging purposes;
more long-term changes would generally be applied to the configuration file
itself. Example usage:
<pre class="programlisting">
$ searchd --index myindex
</pre></li>
</ul></div>
<p>There are some options for <code class="filename">searchd</code> that are specific
to Windows platforms, concerning handling as a service, are only be available on Windows binaries.</p><p>Note that on Windows searchd will default to <code class="option">--console</code> mode, unless you install it as a service.</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--install</code> installs <code class="filename">searchd</code> as a service
into the Microsoft Management Console (Control Panel / Administrative Tools / Services).
Any other parameters specified on the command line, where <code class="option">--install</code>
is specified will also become part of the command line on future starts of the service.
For example, as part of calling <code class="filename">searchd</code>, you will likely also need
to specify the configuration file with <code class="option">--config</code>, and you would do that
as well as specifying <code class="option">--install</code>. Once called, the usual start/stop
facilities will become available via the management console, so any methods you could
use for starting, stopping and restarting services would also apply to
<code class="filename">searchd</code>. Example:
<pre class="programlisting">
C:\WINDOWS\system32&gt; C:\Sphinx\bin\searchd.exe --install
   --config C:\Sphinx\sphinx.conf
</pre>
If you wanted to have the I/O stats every time you started <code class="filename">searchd</code>,
you would specify its option on the same line as the <code class="option">--install</code> command thus:
<pre class="programlisting">
C:\WINDOWS\system32&gt; C:\Sphinx\bin\searchd.exe --install
   --config C:\Sphinx\sphinx.conf --iostats
</pre></li>
<li><code class="option">--delete</code> removes the service from the Microsoft Management Console
and other places where services are registered, after previously installed with
<code class="option">--install</code>. Note, this does not uninstall the software or delete the indexes.
It means the service will not be called from the services systems, and will not be started
on the machine's next start. If currently running as a service, the current instance
will not be terminated (until the next reboot, or <code class="filename">searchd</code> is called
with <code class="option">--stop</code>). If the service was installed with a custom name
(with <code class="option">--servicename</code>), the same name will need to be specified
with <code class="option">--servicename</code> when calling to uninstall. Example:
<pre class="programlisting">
C:\WINDOWS\system32&gt; C:\Sphinx\bin\searchd.exe --delete
</pre></li>
<li><code class="option">--servicename &lt;name&gt;</code> applies the given name to
<code class="filename">searchd</code> when installing or deleting the service, as would appear
in the Management Console; this will default to searchd, but if being deployed on servers
where multiple administrators may log into the system, or a system with multiple
<code class="filename">searchd</code> instances, a more descriptive name may be applicable.
Note that unless combined with <code class="option">--install</code> or <code class="option">--delete</code>,
this option does not do anything. Example:
<pre class="programlisting">
C:\WINDOWS\system32&gt; C:\Sphinx\bin\searchd.exe --install
   --config C:\Sphinx\sphinx.conf --servicename SphinxSearch
</pre></li>
<li><code class="option">--ntservice</code> is the option that is passed by the
Management Console to <code class="filename">searchd</code> to invoke it as a service
on Windows platforms. It would not normally be necessary to call this directly;
this would normally be called by Windows when the service would be started,
although if you wanted to call this as a regular service from the command-line
(as the complement to <code class="option">--console</code>) you could do so in theory.
</li>
</ul></div>
<p>
Last but not least, as every other daemon, <code class="filename">searchd</code> supports a number of signals.
</p><div class="variablelist"><dl><dt><span class="term">SIGTERM</span></dt>
<dd>Initiates a clean shutdown. New queries will not be handled; but queries
	that are already started will not be forcibly interrupted.</dd><dt><span class="term">SIGHUP</span></dt>
<dd>Initiates index rotation. Depending on the value of
	<a href="#conf-seamless-rotate" title="11.4.11.&nbsp;seamless_rotate">seamless_rotate</a> setting,
	new queries might be shortly stalled; clients will receive temporary
	errors.</dd><dt><span class="term">SIGUSR1</span></dt>
<dd>Forces reopen of searchd log and query log files, letting
	you implement log file rotation.</dd></dl></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ref-search"></a>6.3.&nbsp;<code class="filename">search</code> command reference</h2></div></div></div>
<p><code class="filename">search</code> is one of the helper tools within the 
Sphinx package. Whereas <code class="filename">searchd</code> is responsible for 
searches in a server-type environment, <code class="filename">search</code> is 
aimed at testing the index from the command line, and testing the index 
quickly without building a framework to make the connection to the server 
and process its response.</p><p>Note: <code class="filename">search</code> is not intended to be deployed as 
part of a client application; it is strongly recommended you do not write 
an interface to <code class="filename">search</code> instead of 
<code class="filename">searchd</code>, and none of the bundled client APIs support 
this method. (In any event, <code class="filename">search</code> will reload files 
each time, whereas <code class="filename">searchd</code> will cache them in memory 
for performance.)</p><p>That said, many types of query that you could build in the APIs 
could also be made with <code class="filename">search</code>, however for very 
complex searches it may be easier to construct them using a small script 
and the corresponding API. Additionally, some newer features may be 
available in the <code class="filename">searchd</code> system that have not yet 
been brought into <code class="filename">search</code>.</p><p>The calling syntax for <code class="filename">search</code> is as 
follows:</p><pre class="programlisting">
search [OPTIONS] word1 [word2 [word3 [...]]]
</pre><p>When calling <code class="filename">search</code>, it is not necessary to 
have <code class="filename">searchd</code> running; simply that the account running 
<code class="filename">search</code> has read access to the configuration file and 
the location and files of the indexes.</p><p>The default behaviour is to apply a search for word1 (AND word2 AND 
word3... as specified) to all fields in all indexes as given in the 
configuration file. If constructing the equivalent in the API, this would 
be the equivalent to passing <code class="option">SPH_MATCH_ALL</code> to 
<code class="code">SetMatchMode</code>, and specifying <code class="option">*</code> as the 
indexes to query as part of <code class="code">Query</code>.</p><p>There are many options available to <code class="filename">search</code>. 
Firstly, the general options:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--config &lt;file&gt;</code> (<code class="option">-c 
&lt;file&gt;</code> for short) tells <code class="filename">search</code> to use 
the given file as its configuration, just as with 
<code class="filename">indexer</code> above.</li>
<li><code class="option">--index &lt;index&gt;</code> (<code class="option">-i 
&lt;index&gt;</code> for short) tells <code class="filename">search</code> to 
limit searching to the specified index only; normally it would attempt to 
search all of the physical indexes listed in 
<code class="filename">sphinx.conf</code>, not any distributed ones.</li>
<li><code class="option">--stdin</code> tells <code class="filename">search</code> to 
accept the query from the standard input, rather than the command line. 
This can be useful for testing purposes whereby you could feed input via 
pipes and from scripts.</li>
</ul></div>
<p>
</p><p>Options for setting matches:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--any</code> (<code class="option">-a</code> for short) changes 
the matching mode to match any of the words as part of the query (word1 OR 
word2 OR word3). In the API this would be equivalent to passing 
<code class="option">SPH_MATCH_ANY</code> to <code class="code">SetMatchMode</code>.</li>
<li><code class="option">--phrase</code> (<code class="option">-p</code> for short) 
changes the matching mode to match all of the words as part of the query, 
and do so in the phrase given (not including punctuation). In the API this 
would be equivalent to passing <code class="option">SPH_MATCH_PHRASE</code> to 
<code class="code">SetMatchMode</code>.</li>
<li><code class="option">--boolean</code> (<code class="option">-b</code> for short) 
changes the matching mode to <a href="#boolean-syntax" title="5.2.&nbsp;Boolean query syntax">Boolean 
matching</a>. Note if using Boolean syntax matching on the command 
line, you may need to escape the symbols (with a backslash) to avoid the 
shell/command line processor applying them, such as ampersands being 
escaped on a Unix/Linux system to avoid it forking to the 
<code class="filename">search</code> process, although this can be resolved by 
using <code class="option">--stdin</code>, as below. In the API this would be 
equivalent to passing <code class="option">SPH_MATCH_BOOLEAN</code> to 
<code class="code">SetMatchMode</code>.</li>
<li><code class="option">--ext</code> (<code class="option">-e</code> for short) changes 
the matching mode to <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">Extended 
matching</a>. In the API this would be equivalent to passing 
<code class="option">SPH_MATCH_EXTENDED</code> to <code class="code">SetMatchMode</code>, and it 
should be noted that use of this mode is being discouraged in favour of 
Extended2, below.</li>
<li><code class="option">--ext2</code> (<code class="option">-e2</code> for short) changes 
the matching mode to <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">Extended matching, 
version 2</a>. In the API this would be equivalent to passing 
<code class="option">SPH_MATCH_EXTENDED2</code> to <code class="code">SetMatchMode</code>, and it 
should be noted that use of this mode is being recommended in favour of 
Extended, due to being more efficient and providing other 
features.</li>
<li><code class="option">--filter &lt;attr&gt; &lt;v&gt;</code> (<code class="option">-f 
&lt;attr&gt; &lt;v&gt;</code> for short) filters the results such that 
only documents where the attribute given (attr) matches the value given 
(v). For example, <code class="option">--filter deleted 0</code> only matches 
documents with an attribute called 'deleted' where its value is 0. You can 
also add multiple filters on the command line, by specifying multiple 
<code class="option">--filter</code> multiple times, however if you apply a second 
filter to an attribute it will override the first defined 
filter.</li>
</ul></div>
<p>
</p><p>Options for handling the results:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--limit &lt;count&gt;</code> (<code class="option">-l 
count</code> for short) limits the total number of matches back to the 
number given. If a 'group' is specified, this will be the number of 
grouped results. This defaults to 20 results if not specified (as do the 
APIs)</li>
<li><code class="option">--offset &lt;count&gt;</code> (<code class="option">-o 
&lt;count&gt;</code> for short) offsets the result list by the number of 
places set by the count; this would be used for pagination through 
results, where if you have 20 results per 'page', the second page would 
begin at offset 20, the third page at offset 40, etc.</li>
<li><code class="option">--group &lt;attr&gt;</code> (<code class="option">-g 
&lt;attr&gt;</code> for short) specifies that results should be grouped 
together based on the attribute specified. Like the GROUP BY clause in 
SQL, it will combine all results where the attribute given matches, and 
returns a set of results where each returned result is the best from each 
group. Unless otherwise specified, this will be the best match on 
relevance.</li>
<li><code class="option">--groupsort &lt;expr&gt;</code> (<code class="option">-gs 
&lt;expr&gt;</code> for short) instructs that when results are grouped 
with <code class="option">-group</code>, the expression given in &lt;expr&gt; shall 
determine the order of the groups. Note, this does not specify which is 
the best item within the group, only the order in which the groups 
themselves shall be returned.</li>
<li><code class="option">--sortby &lt;clause&gt;</code> (<code class="option">-s 
&lt;clause&gt;</code> for short) specifies that results should be sorted 
in the order listed in &lt;clause&gt;. This allows you to specify the 
order you wish results to be presented in, ordering by different columns. 
For example, you could say <code class="option">--sortby "@weight DESC entrytime 
DESC"</code> to sort entries first by weight (or relevance) and where 
two or more entries have the same weight, to then sort by the time with 
the highest time (newest) first. You will usually need to put the items in 
quotes (<code class="option">--sortby "@weight DESC"</code>) or use commas 
(<code class="option">--sortby @weight,DESC</code>) to avoid the items being treated 
separately. Additionally, like the regular sorting modes, if 
<code class="option">--group</code> (grouping) is being used, this will state how to 
establish the best match within each group.</li>
<li><code class="option">--sortexpr expr</code> (<code class="option">-S expr</code> for 
short) specifies that the search results should be presented in an order 
determined by an arithmetic expression, stated in expr. For example: 
<code class="option">--sortexpr "@weight + ( user_karma + ln(pageviews) )*0.1"</code> 
(again noting that this will have to be quoted to avoid the shell dealing 
with the asterisk). Extended sort mode is discussed in more detail under 
the <code class="option">SPH_SORT_EXTENDED</code> entry under the <a href="#sorting-modes" title="5.5.&nbsp;Sorting modes">Sorting modes</a> section of the 
manual.</li>
<li><code class="option">--sort=date</code> specifies that the results should 
be sorted by descending (i.e. most recent first) date. This requires that 
there is an attribute in the index that is set as a timestamp.</li>
<li><code class="option">--rsort=date</code> specifies that the results should 
be sorted by ascending (i.e. oldest first) date. This requires that there 
is an attribute in the index that is set as a timestamp.</li>
<li><code class="option">--sort=ts</code> specifies that the results should be 
sorted by timestamp in groups; it will return all of the documents whose 
timestamp is within the last hour, then sorted within that bracket for 
relevance. After, it would return the documents from the last day, sorted 
by relevance, then the last week and then the last month. It is discussed 
in more detail under the <code class="option">SPH_SORT_TIME_SEGMENTS</code> entry 
under the <a href="#sorting-modes" title="5.5.&nbsp;Sorting modes">Sorting modes</a> section of 
the manual.</li>
</ul></div>
<p>
</p><p>Other options:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--noinfo</code> (<code class="option">-q</code> for short) 
instructs <code class="filename">search</code> not to look-up data in your SQL 
database. Specifically, for debugging with MySQL and 
<code class="filename">search</code>, you can provide it with a query to look up 
the full article based on the returned document ID. It is explained in 
more detail under the <a href="#conf-sql-query-info" title="11.1.32.&nbsp;sql_query_info">sql_query_info</a> directive.</li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ref-spelldump"></a>6.4.&nbsp;<code class="filename">spelldump</code> command reference</h2></div></div></div>
<p><code class="filename">spelldump</code> is one of the helper tools within the Sphinx package.</p><p>It is used to extract the contents of a dictionary file that uses 
<code class="filename">ispell</code> or <code class="filename">MySpell</code> format, which 
can help build word lists for <em class="glossterm">wordforms</em> - all of 
the possible forms are pre-built for you.</p><p>Its general usage is:</p><pre class="programlisting">
spelldump [options] &lt;dictionary&gt; &lt;affix&gt; [result] [locale-name]
</pre><p>The two main parameters are the dictionary's main file and its affix 
file; usually these are named as 
<code class="filename">[language-prefix].dict</code> and 
<code class="filename">[language-prefix].aff</code> and will be available with most 
common Linux distributions, as well as various places online.</p><p><code class="option">[result]</code> specifies where the dictionary data should 
be output to, and <code class="option">[locale-name]</code> additionally specifies 
the locale details you wish to use.</p><p>There is an additional option, <code class="option">-c [file]</code>, which 
specifies a file for case conversion details.</p><p>Examples of its usage are:</p><pre class="programlisting">
spelldump en.dict en.aff
spelldump ru.dict ru.aff ru.txt ru_RU.CP1251
spelldump ru.dict ru.aff ru.txt .1251
</pre><p>The results file will contain a list of all the words in the 
dictionary in alphabetical order, output in the format of a wordforms file, 
which you can use to customise for your specific circumstances. An example 
of the result file:</p><pre class="programlisting">
zone &gt; zone
zoned &gt; zoned
zoning &gt; zoning
</pre></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ref-indextool"></a>6.5.&nbsp;<code class="filename">indextool</code> command reference</h2></div></div></div>
<p>
<code class="filename">indextool</code> is one of the helper tools within
the Sphinx package, introduced in version 0.9.9-rc2. It is used to
dump miscellaneous debug information about the physical index.
(Additional functionality such as index verification is planned
in the future, hence the indextool name rather than just indexdump.)
Its general usage is:
</p><pre class="programlisting">
indextool &lt;command&gt; [options]
</pre><p>
The only currently available option applies to all commands
and lets you specify the configuration file:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--config &lt;file&gt;</code> (<code class="option">-c &lt;file&gt;</code> for short)
overrides the built-in config file names.
</li>
</ul></div>
<p>
</p><p>
The commands are as follows:
</p><div class="itemizedlist"><ul type="disc"><li><code class="option">--dumpheader FILENAME.sph</code> quickly dumps
the provided index header file without touching any other index files
or even the configuration file. The report provides a breakdown of
all the index settings, in particular the entire attribute and
field list. Prior to 0.9.9-rc2, this command was present in
CLI search utility.
</li>
<li><code class="option">--dumpheader INDEXNAME</code> dumps index header
by index name with looking up the header path in the configuration file.
</li>
<li><code class="option">--dumpdocids INDEXNAME</code> dumps document IDs
by index name. It takes the data from attribute (.spa) file and therefore
requires docinfo=extern to work.
</li>
<li><code class="option">--dumphitlist INDEXNAME KEYWORD</code> dumps all
the hits (occurences) of a given keyword in a given index, with keyword
specified as text.
</li>
<li><code class="option">--dumphitlist INDEXNAME --wordid ID</code> dumps all
the hits (occurences) of a given keyword in a given index, with keyword
specified as internal numeric ID.
</li>
<li><code class="option">--htmlstrip INDEXNAME</code> filters stdin using
HTML stripper settings for a given index, and prints the filtering 
results to stdout. Note that the settings will be taken from sphinx.conf,
and not the index header.
</li>
<li><code class="option">--check INDEXNAME</code> checks the index data
files for consistency errors that might be introduced either by bugs
in <code class="filename">indexer</code> and/or hardware faults.
</li>
<li><code class="option">--strip-path</code> strips the path names from
all the file names referenced from the index (stopwords, wordforms,
exceptions, etc). This is useful for checking indexes built on another
machine with possibly different path layouts.
</li>
</ul></div></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="sphinxql-reference"></a>Chapter&nbsp;7.&nbsp;SphinxQL reference</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#sphinxql-select">7.1. SELECT syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-meta">7.2. SHOW META syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-warnings">7.3. SHOW WARNINGS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-show-status">7.4. SHOW STATUS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-insert">7.5. INSERT and REPLACE syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-delete">7.6. DELETE syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-set">7.7. SET syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-commit">7.8. BEGIN, COMMIT, and ROLLBACK syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-call-snippets">7.9. CALL SNIPPETS syntax</a></span></dt>
<dt><span class="sect1"><a href="#sphinxql-call-keywords">7.10. CALL KEYWORDS syntax</a></span></dt>
</dl></div>
<p>
SphinxQL is our SQL dialect that exposes all of the search daemon
functionality using a standard SQL syntax with a few Sphinx-specific
extensions.  Everything available via the SphinxAPI is also available
SphinxQL but not vice versa; for instance, writes into RT indexes
are only available via SphinxQL.  This chapter documents supported
SphinxQL statements syntax.
</p><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-select"></a>7.1.&nbsp;SELECT syntax</h2></div></div></div>
<pre class="programlisting">
SELECT
    select_expr [, select_expr ...]
    FROM index [, index2 ...]
    [WHERE where_condition]
    [GROUP BY {col_name | expr_alias}]
    [ORDER BY {col_name | expr_alias} {ASC | DESC} [, ...]]
    [WITHIN GROUP ORDER BY {col_name | expr_alias} {ASC | DESC}]
    [LIMIT offset, row_count]
    [OPTION opt_name = opt_value [, ...]]
</pre><p>
<span class="bold"><strong>SELECT</strong></span> statement was introduced in version 0.9.9-rc2.
It's syntax is based upon regular SQL but adds several Sphinx-specific
extensions and has a few omissions (such as (currently) missing support for JOINs).
Specifically,
</p><div class="itemizedlist"><ul type="disc"><li>Column list clause. Column names, arbitrary expressions,
and star ('*') are all allowed (ie. "SELECT @id, group_id*123+456 FROM test1"
will work). Unlike in regular SQL, all computed expressions must be aliased with
a valid identifier. Special names such as @id and @weight should currently
be used with leading at-sign. This at-sign requirement will be lifted in
the future.</li>
<li>FROM clause. FROM clause should contain the list of indexes
to search through. Unlike in regular SQL, comma means enumeration of
full-text indexes as in <a href="#api-func-query" title="8.6.1.&nbsp;Query">Query()</a>
API call rather than JOIN.
</li>
<li>WHERE clause. This clause will map both to fulltext query
and filters. Comparison operators (=, !=, &lt;, &gt;, &lt;=, &gt;=), IN(),
AND, NOT, and BETWEEN are all supported and map directly to filters.
OR is not supported yet but will be in the future. MATCH('query')
is supported and maps to fulltext query. Query will be interpreted
according to <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">full-text query language rules</a>.
There must be at most one MATCH() in the clause.
</li>
<li>GROUP BY clause. Currently only supports grouping by a single
column. The column however can be a computed expression:
<pre class="programlisting">
SELECT *, group_id*1000+article_type AS gkey FROM example GROUP BY gkey
</pre>
Aggregate functions (AVG(), MIN(), MAX(), SUM()) in column list
clause are supported. Arguments to aggregate functions can be either
plain attributes or arbitrary expressions. COUNT(*) is implicitly
supported as using GROUP BY will add @count column to result set.
Explicit support might be added in the future. COUNT(DISTINCT attr)
is supported. Currently there can be at most one COUNT(DISTINCT)
per query and an argument needs to be an attribute. Both current
restrictions on COUNT(DISTINCT) might be lifted in the future.
<pre class="programlisting">
SELECT *, AVG(price) AS avgprice, COUNT(DISTINCT storeid)
FROM products
WHERE MATCH('ipod')
GROUP BY vendorid
</pre></li>
<li>WITHIN GROUP ORDER BY clause. This is a Sphinx specific
extension that lets you control how the best row within a group
will to be selected. The syntax matches that of regular ORDER BY
clause:
<pre class="programlisting">
SELECT *, INTERVAL(posted,NOW()-7*86400,NOW()-86400) AS timeseg
FROM example WHERE MATCH('my search query')
GROUP BY siteid
WITHIN GROUP ORDER BY @weight DESC
ORDER BY timeseg DESC, @weight DESC
</pre></li>
<li>ORDER BY clause. Unlike in regular SQL, only column names
(not expressions) are allowed and explicit ASC and DESC are required.
The columns however can be computed expressions:
<pre class="programlisting">
SELECT *, @weight*10+docboost AS skey FROM example ORDER BY skey
</pre></li>
<li>LIMIT clause. Both LIMIT N and LIMIT M,N forms are supported.
Unlike in regular SQL (but like in Sphinx API), an implicit LIMIT 0,20
is present by default.
</li>
<li>OPTION clause. This is a Sphinx specific extension that
lets you control a number of per-query options. The syntax is:
<pre class="programlisting">
OPTION &lt;optionname&gt;=&lt;value&gt; [ , ... ]
</pre>
Supported options and respectively allowed values are:
<div class="itemizedlist"><ul type="circle"><li>'ranker' - any of 'proximity_bm25', 'bm25', 'none', 'wordcount', 'proximity', 'matchany', or 'fieldmask'</li>
<li>'max_matches' - integer (per-query max matches value)</li>
<li>'cutoff' - integer (max found matches threshold)</li>
<li>'max_query_time' - integer (max search time threshold, msec)</li>
<li>'retry_count' - integer (distributed retries count)</li>
<li>'retry_delay' - integer (distributed retry delay, msec)</li>
<li>'field_weights' - a named integer list (per-field user weights for ranking)</li>
<li>'index_weights' - a named integer list (per-index user weights for ranking)</li>
</ul></div>

Example:
<pre class="programlisting">
SELECT * FROM test WHERE MATCH('@title hello @body world')
OPTION ranker=bm25, max_matches=3000,
    field_weights=(title=10, body=3)
</pre></li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-show-meta"></a>7.2.&nbsp;SHOW META syntax</h2></div></div></div>
<pre class="programlisting">
SHOW META
</pre><p><span class="bold"><strong>SHOW META</strong></span> shows additional meta-information about the latest
query such as query time and keyword statistics:
</p><pre class="programlisting">
mysql&gt; SELECT * FROM test1 WHERE MATCH('test|one|two');
+------+--------+----------+------------+
| id   | weight | group_id | date_added |
+------+--------+----------+------------+
|    1 |   3563 |      456 | 1231721236 |
|    2 |   2563 |      123 | 1231721236 |
|    4 |   1480 |        2 | 1231721236 |
+------+--------+----------+------------+
3 rows in set (0.01 sec)

mysql&gt; SHOW META;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| total         | 3     |
| total_found   | 3     |
| time          | 0.005 |
| keyword[0]    | test  |
| docs[0]       | 3     |
| hits[0]       | 5     |
| keyword[1]    | one   |
| docs[1]       | 1     |
| hits[1]       | 2     |
| keyword[2]    | two   |
| docs[2]       | 1     |
| hits[2]       | 2     |
+---------------+-------+
12 rows in set (0.00 sec)
</pre><p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-show-warnings"></a>7.3.&nbsp;SHOW WARNINGS syntax</h2></div></div></div>
<pre class="programlisting">
SHOW WARNINGS
</pre><p><span class="bold"><strong>SHOW WARNINGS</strong></span> statement, introduced in version 0.9.9-rc2,
can be used to retrieve the warning
produced by the latest query. The error message will be returned along with
the query itself:
</p><pre class="programlisting">
mysql&gt; SELECT * FROM test1 WHERE MATCH('@@title hello') \G
ERROR 1064 (42000): index test1: syntax error, unexpected TOK_FIELDLIMIT
near '@title hello'

mysql&gt; SELECT * FROM test1 WHERE MATCH('@title -hello') \G
ERROR 1064 (42000): index test1: query is non-computable (single NOT operator)

mysql&gt; SELECT * FROM test1 WHERE MATCH('"test doc"/3') \G
*************************** 1. row ***************************
        id: 4
    weight: 2500
  group_id: 2
date_added: 1231721236
1 row in set, 1 warning (0.00 sec)

mysql&gt; SHOW WARNINGS \G
*************************** 1. row ***************************
  Level: warning
   Code: 1000
Message: quorum threshold too high (words=2, thresh=3); replacing quorum operator
         with AND operator
1 row in set (0.00 sec)
</pre><p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-show-status"></a>7.4.&nbsp;SHOW STATUS syntax</h2></div></div></div>
<p><span class="bold"><strong>SHOW STATUS</strong></span>, introduced in version 0.9.9-rc2,
displays a number of useful performance counters. IO and CPU
counters will only be available if searchd was started with --iostats and --cpustats
switches respectively.
</p><pre class="programlisting">
mysql&gt; SHOW STATUS;
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| uptime             | 216   |
| connections        | 3     |
| maxed_out          | 0     |
| command_search     | 0     |
| command_excerpt    | 0     |
| command_update     | 0     |
| command_keywords   | 0     |
| command_persist    | 0     |
| command_status     | 0     |
| agent_connect      | 0     |
| agent_retry        | 0     |
| queries            | 10    |
| dist_queries       | 0     |
| query_wall         | 0.075 |
| query_cpu          | OFF   |
| dist_wall          | 0.000 |
| dist_local         | 0.000 |
| dist_wait          | 0.000 |
| query_reads        | OFF   |
| query_readkb       | OFF   |
| query_readtime     | OFF   |
| avg_query_wall     | 0.007 |
| avg_query_cpu      | OFF   |
| avg_dist_wall      | 0.000 |
| avg_dist_local     | 0.000 |
| avg_dist_wait      | 0.000 |
| avg_query_reads    | OFF   |
| avg_query_readkb   | OFF   |
| avg_query_readtime | OFF   |
+--------------------+-------+
29 rows in set (0.00 sec)
</pre><p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-insert"></a>7.5.&nbsp;INSERT and REPLACE syntax</h2></div></div></div>
<pre class="programlisting">
{INSERT | REPLACE} INTO index [(column, ...)]
	VALUES (value, ...)
	[, (...)]
</pre><p>
INSERT statement, introduced in version 1.10-beta, is only supported for RT indexes.
It inserts new rows (documents) into an existing index, with the provided column values.
</p><p>
ID column must be present in all cases. Rows with duplicate IDs will <span class="bold"><strong>not</strong></span>
be overwritten by INSERT; use REPLACE to do that.
</p><p>
<code class="option">index</code> is the name of RT index into which the new row(s)
should be inserted. The optional column names list lets you only explicitly specify
values for some of the columns present in the index. All the other columns will be
filled with their default values (0 for scalar types, empty string for text types).
</p><p>
Expressions are not currently supported in INSERT and values should be explicitly
specified.
</p><p>
Multiple rows can be inserted using a single INSERT statement by providing
several comma-separated, parens-enclosed lists of rows values.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-delete"></a>7.6.&nbsp;DELETE syntax</h2></div></div></div>
<pre class="programlisting">
DELETE FROM index WHERE id = value
</pre><p>
DELETE statement, introduced in version 1.10-beta, is only supported for RT indexes.
It deletes existing rows (documents) from an existing index based on ID.
</p><p>
<code class="option">index</code> is the name of RT index from which the row should be deleted.
<code class="option">value</code> is the row ID to be deleted.
</p><p>
Additional types of WHERE conditions (such as IN, conditions on attributes, etc)
are planned, but not supported yet as of 1.10-beta.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-set"></a>7.7.&nbsp;SET syntax</h2></div></div></div>
<pre class="programlisting">
SET variable_name = value
</pre><p>
SET statement, introduced in version 1.10-beta, modifies a server variable value.
The only variable supported as of 1.10-beta is AUTOCOMMIT, and its allowed values
are 0 and 1.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-commit"></a>7.8.&nbsp;BEGIN, COMMIT, and ROLLBACK syntax</h2></div></div></div>
<pre class="programlisting">
START TRANSACTION | BEGIN
COMMIT
ROLLBACK
SET AUTOCOMMIT = {0 | 1}
</pre><p>
BEGIN, COMMIT, and ROLLBACK statements were introduced in version 1.10-beta.
BEGIN statement (or its START TRANSACTION alias) forcibly commits pending
transaction, if any, and begins a new one. COMMIT statement commits the current
transaction, making all its changes permanent. ROLLBACK statement rolls back the
current transaction, canceling all its changes. SET AUTOCOMMIT controls the
autocommit mode in the active session.
</p><p>
AUTOCOMMIT is set to 1 by default, meaning that every statement that perfoms
any changes on any index is implicitly wrapped in BEGIN and COMMIT.
</p><p>
Transactions are limited to a single RT index, and also limited in size.
They are atomic, consistent, overly isolated, and durable. Overly isolated
means that the changes are not only invisible to the concurrent transactions
but even to the current session itself.
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-call-snippets"></a>7.9.&nbsp;CALL SNIPPETS syntax</h2></div></div></div>
<pre class="programlisting">
CALL SNIPPETS(data, index, query[, opt_value AS opt_name[, ...]])
</pre><p>
CALL SNIPPETS statement, introduced in version 1.10-beta, builds a snippet
from provided data and query, using specified index settings.
</p><p>
<code class="option">data</code> is the source data string to extract a snippet from.
<code class="option">index</code> is the name of the index from which to take the text
processing settings. <code class="option">query</code> is the full-text query to build
snippets for. Additional options are documented in
<a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">Section&nbsp;8.7.1, &#8220;BuildExcerpts&#8221;</a>. Usage example:
</p><pre class="programlisting">
CALL SNIPPETS('this is my document text', 'test1', 'hello world',
    5 AS around, 200 AS limit)
</pre></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxql-call-keywords"></a>7.10.&nbsp;CALL KEYWORDS syntax</h2></div></div></div>
<pre class="programlisting">
CALL KEYWORDS(text, index, [hits])
</pre><p>
CALL KEYWORDS statement, introduced in version 1.10-beta, splits text
into particular keywords. It returns tokenized and normalized forms
of the keywords, and, optionally, keyword statistics.
</p><p>
<code class="option">text</code> is the text to break down to keywords.
<code class="option">index</code> is the name of the index from which to take the text
processing settings. <code class="option">hits</code> is an optional boolean parameter
that specifies whether to return document and hit occurrence statistics.
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="api-reference"></a>Chapter&nbsp;8.&nbsp;API reference</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#api-funcgroup-general">8.1. General API functions</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-getlasterror">8.1.1. GetLastError</a></span></dt>
<dt><span class="sect2"><a href="#api-func-getlastwarning">8.1.2. GetLastWarning</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setserver">8.1.3. SetServer</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setretries">8.1.4. SetRetries</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setconnecttimeout">8.1.5. SetConnectTimeout</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setarrayresult">8.1.6. SetArrayResult</a></span></dt>
<dt><span class="sect2"><a href="#api-func-isconnecterror">8.1.7. IsConnectError</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-general-query-settings">8.2. General query settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setlimits">8.2.1. SetLimits</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setmaxquerytime">8.2.2. SetMaxQueryTime</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setoverride">8.2.3. SetOverride</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setselect">8.2.4. SetSelect</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-fulltext-query-settings">8.3. Full-text search query settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setmatchmode">8.3.1. SetMatchMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setrankingmode">8.3.2. SetRankingMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setsortmode">8.3.3. SetSortMode</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setweights">8.3.4. SetWeights</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfieldweights">8.3.5. SetFieldWeights</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setindexweights">8.3.6. SetIndexWeights</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-filtering">8.4. Result set filtering settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setidrange">8.4.1. SetIDRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilter">8.4.2. SetFilter</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilterrange">8.4.3. SetFilterRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setfilterfloatrange">8.4.4. SetFilterFloatRange</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setgeoanchor">8.4.5. SetGeoAnchor</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-groupby">8.5. GROUP BY settings</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-setgroupby">8.5.1. SetGroupBy</a></span></dt>
<dt><span class="sect2"><a href="#api-func-setgroupdistinct">8.5.2. SetGroupDistinct</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-querying">8.6. Querying</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-query">8.6.1. Query</a></span></dt>
<dt><span class="sect2"><a href="#api-func-addquery">8.6.2. AddQuery</a></span></dt>
<dt><span class="sect2"><a href="#api-func-runqueries">8.6.3. RunQueries</a></span></dt>
<dt><span class="sect2"><a href="#api-func-resetfilters">8.6.4. ResetFilters</a></span></dt>
<dt><span class="sect2"><a href="#api-func-resetgroupby">8.6.5. ResetGroupBy</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-additional-functionality">8.7. Additional functionality</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-buildexcerpts">8.7.1. BuildExcerpts</a></span></dt>
<dt><span class="sect2"><a href="#api-func-updateatttributes">8.7.2. UpdateAttributes</a></span></dt>
<dt><span class="sect2"><a href="#api-func-buildkeywords">8.7.3. BuildKeywords</a></span></dt>
<dt><span class="sect2"><a href="#api-func-escapestring">8.7.4. EscapeString</a></span></dt>
<dt><span class="sect2"><a href="#api-func-status">8.7.5. Status</a></span></dt>
<dt><span class="sect2"><a href="#api-func-flushattributes">8.7.6. FlushAttributes</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#api-funcgroup-pconn">8.8. Persistent connections</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#api-func-open">8.8.1. Open</a></span></dt>
<dt><span class="sect2"><a href="#api-func-close">8.8.2. Close</a></span></dt>
</dl></dd></dl></div>
<p>
There is a number of native searchd client API implementations
for Sphinx. As of time of this writing, we officially support our own
PHP, Python, and Java implementations. There also are third party
free, open-source API implementations for Perl, Ruby, and C++.
</p><p>
The reference API implementation is in PHP, because (we believe)
Sphinx is most widely used with PHP than any other language.
This reference documentation is in turn based on reference PHP API,
and all code samples in this section will be given in PHP.
</p><p>
However, all other APIs provide the same methods and implement
the very same network protocol. Therefore the documentation does
apply to them as well. There might be minor differences as to the
method naming conventions or specific data structures used.
But the provided functionality must not differ across languages.
</p><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-general"></a>8.1.&nbsp;General API functions</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-getlasterror"></a>8.1.1.&nbsp;GetLastError</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function GetLastError()</p><p>
Returns last error message, as a string, in human readable format.
If there were no errors during the previous API call, empty string is returned.
</p><p>
You should call it when any other function (such as <a href="#api-func-query" title="8.6.1.&nbsp;Query">Query()</a>)
fails (typically, the failing function returns false). The returned string will
contain the error description.
</p><p>
The error message is <span class="emphasis"><em>not</em></span> reset by this call; so you can safely
call it several times if needed.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-getlastwarning"></a>8.1.2.&nbsp;GetLastWarning</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function GetLastWarning ()</p><p>
Returns last warning message, as a string, in human readable format.
If there were no warnings during the previous API call, empty string is returned.
</p><p>
You should call it to verify whether your request
(such as <a href="#api-func-query" title="8.6.1.&nbsp;Query">Query()</a>) was completed but with warnings.
For instance, search query against a distributed index might complete
succesfully even if several remote agents timed out. In that case,
a warning message would be produced.
</p><p>
The warning message is <span class="emphasis"><em>not</em></span> reset by this call; so you can safely
call it several times if needed.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setserver"></a>8.1.3.&nbsp;SetServer</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetServer ( $host, $port )</p><p>
Sets <code class="filename">searchd</code> host name and TCP port.
All subsequent requests will use the new host and port settings.
Default host and port are 'localhost' and 9312, respectively.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setretries"></a>8.1.4.&nbsp;SetRetries</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetRetries ( $count, $delay=0 )</p><p>
Sets distributed retry count and delay.
</p><p>
On temporary failures <code class="filename">searchd</code> will attempt up to
<code class="code">$count</code> retries per agent. <code class="code">$delay</code> is the delay
between the retries, in milliseconds. Retries are disabled by default.
Note that this call will <span class="bold"><strong>not</strong></span> make the API itself retry on
temporary failure; it only tells <code class="filename">searchd</code> to do so.
Currently, the list of temporary failures includes all kinds of connect()
failures and maxed out (too busy) remote agents.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setconnecttimeout"></a>8.1.5.&nbsp;SetConnectTimeout</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetConnectTimeout ( $timeout )</p><p>
Sets the time allowed to spend connecting to the server before giving up.
</p><p>Under some circumstances, the server can be delayed in responding, either
due to network delays, or a query backlog. In either instance, this allows
the client application programmer some degree of control over how their
program interacts with <code class="filename">searchd</code> when not available,
and can ensure that the client application does not fail due to exceeding
the script execution limits (especially in PHP).
</p><p>In the event of a failure to connect, an appropriate error code should
be returned back to the application in order for application-level error handling
to advise the user.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setarrayresult"></a>8.1.6.&nbsp;SetArrayResult</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetArrayResult ( $arrayresult )</p><p>
PHP specific. Controls matches format in the search results set
(whether matches should be returned as an array or a hash).
</p><p>
<code class="code">$arrayresult</code> argument must be boolean. If <code class="code">$arrayresult</code> is <code class="code">false</code>
(the default mode), matches will returned in PHP hash format with
document IDs as keys, and other information (weight, attributes)
as values. If <code class="code">$arrayresult</code> is true, matches will be returned
as a plain array with complete per-match information including
document ID.
</p><p>
Introduced along with GROUP BY support on MVA attributes.
Group-by-MVA result sets may contain duplicate document IDs.
Thus they need to be returned as plain arrays, because hashes
will only keep one entry per document ID.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-isconnecterror"></a>8.1.7.&nbsp;IsConnectError</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function IsConnectError ()</p><p>
Checks whether the last error was a network error on API side, or a remote error
reported by searchd. Returns true if the last connection attempt to searchd failed on API side,
false otherwise (if the error was remote, or there were no connection attempts at all).
Introduced in version 0.9.9-rc1.
</p></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-general-query-settings"></a>8.2.&nbsp;General query settings</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setlimits"></a>8.2.1.&nbsp;SetLimits</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetLimits ( $offset, $limit, $max_matches=0, $cutoff=0 )</p><p>
Sets offset into server-side result set (<code class="code">$offset</code>) and amount of matches
to return to client starting from that offset (<code class="code">$limit</code>). Can additionally
control maximum server-side result set size for current query (<code class="code">$max_matches</code>)
and the threshold amount of matches to stop searching at (<code class="code">$cutoff</code>).
All parameters must be non-negative integers.
</p><p>
First two parameters to SetLimits() are identical in behavior to MySQL
LIMIT clause. They instruct <code class="filename">searchd</code> to return at
most <code class="code">$limit</code> matches starting from match number <code class="code">$offset</code>.
The default offset and limit settings are 0 and 20, that is, to return
first 20 matches.
</p><p>
<code class="code">max_matches</code> setting controls how much matches <code class="filename">searchd</code>
will keep in RAM while searching. <span class="bold"><strong>All</strong></span> matching documents will be normally
processed, ranked, filtered, and sorted even if <code class="code">max_matches</code> is set to 1.
But only best N documents are stored in memory at any given moment for performance
and RAM usage reasons, and this setting controls that N. Note that there are
<span class="bold"><strong>two</strong></span> places where <code class="code">max_matches</code> limit is enforced. Per-query
limit is controlled by this API call, but there also is per-server limit
controlled by <code class="code">max_matches</code> setting in the config file. To prevent
RAM usage abuse, server will not allow to set per-query limit
higher than the per-server limit.
</p><p>
You can't retrieve more than <code class="code">max_matches</code> matches to the client application.
The default limit is set to 1000. Normally, you must not have to go over
this limit. One thousand records is enough to present to the end user.
And if you're thinking about pulling the results to application
for further sorting or filtering, that would be <span class="bold"><strong>much</strong></span> more efficient
if performed on Sphinx side.
</p><p>
<code class="code">$cutoff</code> setting is intended for advanced performance control.
It tells <code class="filename">searchd</code> to forcibly stop search query
once <code class="code">$cutoff</code> matches had been found and processed.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setmaxquerytime"></a>8.2.2.&nbsp;SetMaxQueryTime</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetMaxQueryTime ( $max_query_time )</p><p>
Sets maximum search query time, in milliseconds. Parameter must be
a non-negative integer. Default valus is 0 which means "do not limit".
</p><p>Similar to <code class="code">$cutoff</code> setting from <a href="#api-func-setlimits" title="8.2.1.&nbsp;SetLimits">SetLimits()</a>,
but limits elapsed query time instead of processed matches count. Local search queries
will be stopped once that much time has elapsed. Note that if you're performing
a search which queries several local indexes, this limit applies to each index
separately.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setoverride"></a>8.2.3.&nbsp;SetOverride</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetOverride ( $attrname, $attrtype, $values )</p><p>
Sets temporary (per-query) per-document attribute value overrides.
Only supports scalar attributes. $values must be a hash that maps document
IDs to overridden attribute values. Introduced in version 0.9.9-rc1.
</p><p>
Override feature lets you "temporary" update attribute values for some documents
within a single query, leaving all other queries unaffected. This might be useful
for personalized data. For example, assume you're implementing a personalized
search function that wants to boost the posts that the user's friends recommend.
Such data is not just dynamic, but also personal; so you can't simply put it
in the index because you don't want everyone's searches affected. Overrides,
on the other hand, are local to a single query and invisible to everyone else.
So you can, say, setup a "friends_weight" value for every document, defaulting to 0,
then temporary override it with 1 for documents 123, 456 and 789 (recommended by
exactly the friends of current user), and use that value when ranking.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setselect"></a>8.2.4.&nbsp;SetSelect</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetSelect ( $clause )</p><p>
Sets the select clause, listing specific attributes to fetch, and <a href="#sort-expr">expressions</a>
to compute and fetch. Clause syntax mimics SQL. Introduced in version 0.9.9-rc1.</p><p>
SetSelect() is very similar to the part of a typical SQL query between SELECT and FROM.
It lets you choose what attributes (columns) to fetch, and also what expressions
over the columns to compute and fetch. A certain difference from SQL is that expressions
<span class="bold"><strong>must</strong></span> always be aliased to a correct identifier (consisting of letters and digits)
using 'AS' keyword. SQL also lets you do that but does not require to. Sphinx enforces
aliases so that the computation results can always be returned under a "normal" name
in the result set, used in other clauses, etc. 
</p><p>
Everything else is basically identical to SQL. Star ('*') is supported.
Functions are supported. Arbitrary amount of expressions is supported.
Computed expressions can be used for sorting, filtering, and grouping,
just as the regular attributes.
</p><p>
Starting with version 0.9.9-rc2, aggregate functions (AVG(), MIN(),
MAX(), SUM()) are supported when using GROUP BY.
</p><p>
Expression sorting (<a href="#sort-expr">Section&nbsp;5.5, &#8220;SPH_SORT_EXPR mode&#8221;</a>) and geodistance functions
(<a href="#api-func-setgeoanchor" title="8.4.5.&nbsp;SetGeoAnchor">Section&nbsp;8.4.5, &#8220;SetGeoAnchor&#8221;</a>) are now internally implemented using
this computed expressions mechanism, using magic names '@expr' and '@geodist'
respectively.
</p><h4>Example:</h4><pre class="programlisting">
$cl-&gt;SetSelect ( "*, @weight+(user_karma+ln(pageviews))*0.1 AS myweight" );
$cl-&gt;SetSelect ( "exp_years, salary_gbp*{$gbp_usd_rate} AS salary_usd,
   IF(age&gt;40,1,0) AS over40" );
$cl-&gt;SetSelect ( "*, AVG(price) AS avgprice" );
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-fulltext-query-settings"></a>8.3.&nbsp;Full-text search query settings</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setmatchmode"></a>8.3.1.&nbsp;SetMatchMode</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetMatchMode ( $mode )</p><p>
Sets full-text query matching mode, as described in <a href="#matching-modes" title="5.1.&nbsp;Matching modes">Section&nbsp;5.1, &#8220;Matching modes&#8221;</a>.
Parameter must be a constant specifying one of the known modes.
</p><p>
<span class="bold"><strong>WARNING:</strong></span> (PHP specific) you <span class="bold"><strong>must not</strong></span> take the matching mode
constant name in quotes, that syntax specifies a string and is incorrect:
</p><pre class="programlisting">
$cl-&gt;SetMatchMode ( "SPH_MATCH_ANY" ); // INCORRECT! will not work as expected
$cl-&gt;SetMatchMode ( SPH_MATCH_ANY ); // correct, works OK
</pre><p>
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setrankingmode"></a>8.3.2.&nbsp;SetRankingMode</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetRankingMode ( $ranker )</p><p>
Sets ranking mode. Only available in SPH_MATCH_EXTENDED2 matching
mode at the time of this writing. Parameter must be a constant
specifying one of the known modes.
</p><p>
By default, Sphinx computes two factors which contribute to the final
match weight. The major part is query phrase proximity to document text.
The minor part is so-called BM25 statistical function, which varies
from 0 to 1 depending on the keyword frequency within document
(more occurrences yield higher weight) and within the whole index
(more rare keywords yield higher weight).
</p><p>
However, in some cases you'd want to compute weight differently -
or maybe avoid computing it at all for performance reasons because
you're sorting the result set by something else anyway. This can be
accomplished by setting the appropriate ranking mode.
</p><p>
Currently implemented modes are:
</p><div class="itemizedlist"><ul type="disc"><li>
SPH_RANK_PROXIMITY_BM25, default ranking mode which uses and combines
both phrase proximity and BM25 ranking.
</li>
<li>
SPH_RANK_BM25, statistical ranking mode which uses BM25 ranking only (similar to
most other full-text engines). This mode is faster but may result in worse quality
on queries which contain more than 1 keyword.
</li>
<li>
SPH_RANK_NONE, disabled ranking mode. This mode is the fastest.
It is essentially equivalent to boolean searching. A weight of 1 is assigned
to all matches.
</li>
<li>SPH_RANK_WORDCOUNT, ranking by keyword occurrences count. This ranker
computes the amount of per-field keyword occurrences, then multiplies the
amounts by field weights, then sums the resulting values for the final result.
</li>
<li>
SPH_RANK_PROXIMITY, added in version 0.9.9-rc1, returns raw phrase proximity
value as a result. This mode is internally used to emulate SPH_MATCH_ALL queries.
</li>
<li>
SPH_RANK_MATCHANY, added in version 0.9.9-rc1, returns rank as it was computed
in SPH_MATCH_ANY mode ealier, and is internally used to emulate SPH_MATCH_ANY queries.
</li>
<li>
SPH_RANK_FIELDMASK, added in version 0.9.9-rc2, returns a 32-bit mask with
N-th bit corresponding to N-th fulltext field, numbering from 0. The bit will
only be set when the respective field has any keyword occurences satisfiying
the query.
</li>
<li>
SPH_RANK_SPH04, added in version 1.10-beta, is generally based on the default
SPH_RANK_PROXIMITY_BM25 ranker, but additionally boosts the matches when
they occur in the very beginning or the very end of a text field. Thus,
if a field equals the exact query, SPH04 should rank it higher than a field
that contains the exact query but is not equal to it. (For instance, when
the query is "Hyde Park", a document entitled "Hyde Park" should be ranked
higher than a one entitled "Hyde Park, London" or "The Hyde Park Cafe".)
</li>
</ul></div>
<p>
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setsortmode"></a>8.3.3.&nbsp;SetSortMode</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetSortMode ( $mode, $sortby="" )</p><p>
Set matches sorting mode, as described in <a href="#sorting-modes" title="5.5.&nbsp;Sorting modes">Section&nbsp;5.5, &#8220;Sorting modes&#8221;</a>.
Parameter must be a constant specifying one of the known modes.
</p><p>
<span class="bold"><strong>WARNING:</strong></span> (PHP specific) you <span class="bold"><strong>must not</strong></span> take the matching mode
constant name in quotes, that syntax specifies a string and is incorrect:
</p><pre class="programlisting">
$cl-&gt;SetSortMode ( "SPH_SORT_ATTR_DESC" ); // INCORRECT! will not work as expected
$cl-&gt;SetSortMode ( SPH_SORT_ATTR_ASC ); // correct, works OK
</pre><p>
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setweights"></a>8.3.4.&nbsp;SetWeights</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetWeights ( $weights )</p><p>
Binds per-field weights in the order of appearance in the index.
<span class="bold"><strong>DEPRECATED</strong></span>, use <a href="#api-func-setfieldweights" title="8.3.5.&nbsp;SetFieldWeights">SetFieldWeights()</a> instead.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setfieldweights"></a>8.3.5.&nbsp;SetFieldWeights</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetFieldWeights ( $weights )</p><p>
Binds per-field weights by name. Parameter must be a hash (associative array)
mapping string field names to integer weights.
</p><p>
Match ranking can be affected by per-field weights. For instance,
see <a href="#weighting" title="5.4.&nbsp;Weighting">Section&nbsp;5.4, &#8220;Weighting&#8221;</a> for an explanation how phrase proximity
ranking is affected. This call lets you specify what non-default
weights to assign to different full-text fields.
</p><p>
The weights must be positive 32-bit integers. The final weight
will be a 32-bit integer too. Default weight value is 1. Unknown
field names will be silently ignored.
</p><p>
There is no enforced limit on the maximum weight value at the
moment. However, beware that if you set it too high you can start
hitting 32-bit wraparound issues. For instance, if you set
a weight of 10,000,000 and search in extended mode, then
maximum possible weight will be equal to 10 million (your weight)
by 1 thousand (internal BM25 scaling factor, see <a href="#weighting" title="5.4.&nbsp;Weighting">Section&nbsp;5.4, &#8220;Weighting&#8221;</a>)
by 1 or more (phrase proximity rank). The result is at least 10 billion
that does not fit in 32 bits and will be wrapped around, producing
unexpected results.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setindexweights"></a>8.3.6.&nbsp;SetIndexWeights</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetIndexWeights ( $weights )</p><p>
Sets per-index weights, and enables weighted summing of match weights
across different indexes. Parameter must be a hash (associative array)
mapping string index names to integer weights. Default is empty array
that means to disable weighting summing.
</p><p>
When a match with the same document ID is found in several different
local indexes, by default Sphinx simply chooses the match from the index
specified last in the query. This is to support searching through
partially overlapping index partitions.
</p><p>
However in some cases the indexes are not just partitions, and you
might want to sum the weights across the indexes instead of picking one.
<code class="code">SetIndexWeights()</code> lets you do that. With summing enabled,
final match weight in result set will be computed as a sum of match
weight coming from the given index multiplied by respective per-index
weight specified in this call. Ie. if the document 123 is found in
index A with the weight of 2, and also in index B with the weight of 3,
and you called <code class="code">SetIndexWeights ( array ( "A"=&gt;100, "B"=&gt;10 ) )</code>,
the final weight return to the client will be 2*100+3*10 = 230.
</p></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-filtering"></a>8.4.&nbsp;Result set filtering settings</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setidrange"></a>8.4.1.&nbsp;SetIDRange</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetIDRange ( $min, $max )</p><p>
Sets an accepted range of document IDs. Parameters must be integers.
Defaults are 0 and 0; that combination means to not limit by range.
</p><p>
After this call, only those records that have document ID
between <code class="code">$min</code> and <code class="code">$max</code> (including IDs
exactly equal to <code class="code">$min</code> or <code class="code">$max</code>)
will be matched.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setfilter"></a>8.4.2.&nbsp;SetFilter</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetFilter ( $attribute, $values, $exclude=false )</p><p>
Adds new integer values set filter. 
</p><p>
On this call, additional new filter is added to the existing
list of filters. <code class="code">$attribute</code> must be a string with
attribute name. <code class="code">$values</code> must be a plain array
containing integer values. <code class="code">$exclude</code> must be a boolean
value; it controls whether to accept the matching documents
(default mode, when <code class="code">$exclude</code> is false) or reject them.
</p><p>
Only those documents where <code class="code">$attribute</code> column value
stored in the index matches any of the values from <code class="code">$values</code>
array will be matched (or rejected, if <code class="code">$exclude</code> is true).
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setfilterrange"></a>8.4.3.&nbsp;SetFilterRange</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetFilterRange ( $attribute, $min, $max, $exclude=false )</p><p>
Adds new integer range filter.
</p><p>
On this call, additional new filter is added to the existing
list of filters. <code class="code">$attribute</code> must be a string with
attribute name. <code class="code">$min</code> and <code class="code">$max</code> must be
integers that define the acceptable attribute values range
(including the boundaries). <code class="code">$exclude</code> must be a boolean
value; it controls whether to accept the matching documents
(default mode, when <code class="code">$exclude</code> is false) or reject them.
</p><p>
Only those documents where <code class="code">$attribute</code> column value
stored in the index is between <code class="code">$min</code> and <code class="code">$max</code>
(including values that are exactly equal to <code class="code">$min</code> or <code class="code">$max</code>)
will be matched (or rejected, if <code class="code">$exclude</code> is true).
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setfilterfloatrange"></a>8.4.4.&nbsp;SetFilterFloatRange</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetFilterFloatRange ( $attribute, $min, $max, $exclude=false )</p><p>
Adds new float range filter.
</p><p>
On this call, additional new filter is added to the existing
list of filters. <code class="code">$attribute</code> must be a string with
attribute name. <code class="code">$min</code> and <code class="code">$max</code> must be
floats that define the acceptable attribute values range
(including the boundaries). <code class="code">$exclude</code> must be a boolean
value; it controls whether to accept the matching documents
(default mode, when <code class="code">$exclude</code> is false) or reject them.
</p><p>
Only those documents where <code class="code">$attribute</code> column value
stored in the index is between <code class="code">$min</code> and <code class="code">$max</code>
(including values that are exactly equal to <code class="code">$min</code> or <code class="code">$max</code>)
will be matched (or rejected, if <code class="code">$exclude</code> is true).
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setgeoanchor"></a>8.4.5.&nbsp;SetGeoAnchor</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetGeoAnchor ( $attrlat, $attrlong, $lat, $long )</p><p>
Sets anchor point for and geosphere distance (geodistance) calculations, and enable them.
</p><p>
<code class="code">$attrlat</code> and <code class="code">$attrlong</code> must be strings that contain the names
of latitude and longitude attributes, respectively. <code class="code">$lat</code> and <code class="code">$long</code>
are floats that specify anchor point latitude and longitude, in radians.
</p><p>
Once an anchor point is set, you can use magic <code class="code">"@geodist"</code> attribute
name in your filters and/or sorting expressions. Sphinx will compute geosphere distance
between the given anchor point and a point specified by latitude and lognitude
attributes from each full-text match, and attach this value to the resulting match.
The latitude and longitude values both in <code class="code">SetGeoAnchor</code> and the index
attribute data are expected to be in radians. The result will be returned in meters,
so geodistance value of 1000.0 means 1 km. 1 mile is approximately 1609.344 meters.
</p></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-groupby"></a>8.5.&nbsp;GROUP BY settings</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setgroupby"></a>8.5.1.&nbsp;SetGroupBy</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetGroupBy ( $attribute, $func, $groupsort="@group desc" )</p><p>
Sets grouping attribute, function, and groups sorting mode; and enables grouping
(as described in <a href="#clustering" title="5.6.&nbsp;Grouping (clustering) search results ">Section&nbsp;5.6, &#8220;Grouping (clustering) search results &#8221;</a>).
</p><p>
<code class="code">$attribute</code> is a string that contains group-by attribute name.
<code class="code">$func</code> is a constant that chooses a function applied to the attribute value in order to compute group-by key.
<code class="code">$groupsort</code> is a clause that controls how the groups will be sorted. Its syntax is similar
to that described in <a href="#sort-extended">Section&nbsp;5.5, &#8220;SPH_SORT_EXTENDED mode&#8221;</a>.
</p><p>
Grouping feature is very similar in nature to GROUP BY clause from SQL.
Results produces by this function call are going to be the same as produced
by the following pseudo code:
</p><pre class="programlisting">
SELECT ... GROUP BY $func($attribute) ORDER BY $groupsort
</pre><p>
Note that it's <code class="code">$groupsort</code> that affects the order of matches
in the final result set. Sorting mode (see <a href="#api-func-setsortmode" title="8.3.3.&nbsp;SetSortMode">Section&nbsp;8.3.3, &#8220;SetSortMode&#8221;</a>)
affect the ordering of matches <span class="emphasis"><em>within</em></span> group, ie.
what match will be selected as the best one from the group.
So you can for instance order the groups by matches count
and select the most relevant match within each group at the same time.
</p><p>
Starting with version 0.9.9-rc2, aggregate functions (AVG(), MIN(),
MAX(), SUM()) are supported through <a href="#api-func-setselect" title="8.2.4.&nbsp;SetSelect">SetSelect()</a> API call
when using GROUP BY.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-setgroupdistinct"></a>8.5.2.&nbsp;SetGroupDistinct</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function SetGroupDistinct ( $attribute )</p><p>
Sets attribute name for per-group distinct values count calculations.
Only available for grouping queries.
</p><p>
<code class="code">$attribute</code> is a string that contains the attribute name.
For each group, all values of this attribute will be stored (as RAM limits
permit), then the amount of distinct values will be calculated and returned
to the client. This feature is similar to <code class="code">COUNT(DISTINCT)</code>
clause in standard SQL; so these Sphinx calls:
</p><pre class="programlisting">
$cl-&gt;SetGroupBy ( "category", SPH_GROUPBY_ATTR, "@count desc" );
$cl-&gt;SetGroupDistinct ( "vendor" );
</pre><p>
can be expressed using the following SQL clauses:
</p><pre class="programlisting">
SELECT id, weight, all-attributes,
	COUNT(DISTINCT vendor) AS @distinct,
	COUNT(*) AS @count
FROM products
GROUP BY category
ORDER BY @count DESC
</pre><p>
In the sample pseudo code shown just above, <code class="code">SetGroupDistinct()</code> call
corresponds to <code class="code">COUNT(DISINCT vendor)</code> clause only.
<code class="code">GROUP BY</code>, <code class="code">ORDER BY</code>, and  <code class="code">COUNT(*)</code>
clauses are all an equivalent of <code class="code">SetGroupBy()</code> settings. Both queries
will return one matching row for each category. In addition to indexed attributes,
matches will also contain total per-category matches count, and the count
of distinct vendor IDs within each category.
</p></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-querying"></a>8.6.&nbsp;Querying</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-query"></a>8.6.1.&nbsp;Query</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function Query ( $query, $index="*", $comment="" )</p><p>
Connects to <code class="filename">searchd</code> server, runs given search query
with current settings, obtains and returns the result set.
</p><p>
<code class="code">$query</code> is a query string. <code class="code">$index</code> is an index name (or names) string.
Returns false and sets <code class="code">GetLastError()</code> message on general error. 
Returns search result set on success.
Additionally, the contents of <code class="code">$comment</code> are sent to the query log, marked in square brackets, just before the search terms, which can be very useful for debugging.

Currently, the comment is limited to 128 characters.
</p><p>
Default value for <code class="code">$index</code> is <code class="code">"*"</code> that means
to query all local indexes. Characters allowed in index names include
Latin letters (a-z), numbers (0-9), minus sign (-), and underscore (_);
everything else is considered a separator. Therefore, all of the
following samples calls are valid and will search the same
two indexes:
</p><pre class="programlisting">
$cl-&gt;Query ( "test query", "main delta" );
$cl-&gt;Query ( "test query", "main;delta" );
$cl-&gt;Query ( "test query", "main, delta" );
</pre><p>
Index specification order matters. If document with identical IDs are found
in two or more indexes, weight and attribute values from the very last matching 
index will be used for sorting and returning to client (unless explicitly
overridden with <a href="#api-func-setindexweights" title="8.3.6.&nbsp;SetIndexWeights">SetIndexWeights()</a>). Therefore,
in the example above, matches from "delta" index will always win over
matches from "main".
</p><p>
On success, <code class="code">Query()</code> returns a result set that contains
some of the found matches (as requested by <a href="#api-func-setlimits" title="8.2.1.&nbsp;SetLimits">SetLimits()</a>)
and additional general per-query statistics. The result set is a hash
(PHP specific; other languages might utilize other structures instead
of hash) with the following keys and values:
</p><div class="variablelist"><dl><dt><span class="term">"matches":</span></dt>
<dd>Hash which maps found document IDs to another small hash containing document weight and attribute values
		(or an array of the similar small hashes if <a href="#api-func-setarrayresult" title="8.1.6.&nbsp;SetArrayResult">SetArrayResult()</a> was enabled).
	</dd><dt><span class="term">"total":</span></dt>
<dd>Total amount of matches retrieved <span class="emphasis"><em>on server</em></span> (ie. to the server side result set) by this query.
		You can retrieve up to this amount of matches from server for this query text with current query settings.
	</dd><dt><span class="term">"total_found":</span></dt>
<dd>Total amount of matching documents in index (that were found and procesed on server).</dd><dt><span class="term">"words":</span></dt>
<dd>Hash which maps query keywords (case-folded, stemmed, and otherwise processed) to a small hash with per-keyword statitics ("docs", "hits").</dd><dt><span class="term">"error":</span></dt>
<dd>Query error message reported by <code class="filename">searchd</code> (string, human readable). Empty if there were no errors.</dd><dt><span class="term">"warning":</span></dt>
<dd>Query warning message reported by <code class="filename">searchd</code> (string, human readable). Empty if there were no warnings.</dd></dl></div>
<p>
</p><p>
It should be noted that <code class="code">Query()</code> carries out the same actions as
<code class="code">AddQuery()</code> and <code class="code">RunQueries()</code> without the intermediate steps;
it is analoguous to a single <code class="code">AddQuery()</code> call, followed by a corresponding
<code class="code">RunQueries()</code>, then returning the first array element of matches
(from the first, and only, query.)
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-addquery"></a>8.6.2.&nbsp;AddQuery</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function AddQuery ( $query, $index="*", $comment="" )</p><p>
Adds additional query with current settings to multi-query batch.
<code class="code">$query</code> is a query string. <code class="code">$index</code> is an index name (or names) string.
Additionally if provided, the contents of <code class="code">$comment</code> are sent to the query log,
marked in square brackets, just before the search terms, which can be very useful for debugging.
Currently, this is limited to 128 characters.
Returns index to results array returned from <a href="#api-func-runqueries" title="8.6.3.&nbsp;RunQueries">RunQueries()</a>.
</p><p>
Batch queries (or multi-queries) enable <code class="filename">searchd</code> to perform internal
optimizations if possible. They also reduce network connection overheads and search process
creation overheads in all cases. They do not result in any additional overheads compared
to simple queries. Thus, if you run several different queries from your web page,
you should always consider using multi-queries.
</p><p>
For instance, running the same full-text query but with different
sorting or group-by settings will enable <code class="filename">searchd</code>
to perform expensive full-text search and ranking operation only once,
but compute multiple group-by results from its output.
</p><p>
This can be a big saver when you need to display not just plain
search results but also some per-category counts, such as the amount of
products grouped by vendor. Without multi-query, you would have to run several
queries which perform essentially the same search and retrieve the
same matches, but create result sets differently. With multi-query,
you simply pass all these querys in a single batch and Sphinx
optimizes the redundant full-text search internally.
</p><p>
<code class="code">AddQuery()</code> internally saves full current settings state
along with the query, and you can safely change them afterwards for subsequent
<code class="code">AddQuery()</code> calls. Already added queries will not be affected;
there's actually no way to change them at all. Here's an example:
</p><pre class="programlisting">
$cl-&gt;SetSortMode ( SPH_SORT_RELEVANCE );
$cl-&gt;AddQuery ( "hello world", "documents" );

$cl-&gt;SetSortMode ( SPH_SORT_ATTR_DESC, "price" );
$cl-&gt;AddQuery ( "ipod", "products" );

$cl-&gt;AddQuery ( "harry potter", "books" );

$results = $cl-&gt;RunQueries ();
</pre><p>
With the code above, 1st query will search for "hello world" in "documents" index
and sort results by relevance, 2nd query will search for "ipod" in "products"
index and sort results by price, and 3rd query will search for "harry potter"
in "books" index while still sorting by price. Note that 2nd <code class="code">SetSortMode()</code> call
does not affect the first query (because it's already added) but affects both other
subsequent queries.
</p><p>
Additionally, any filters set up before an <code class="code">AddQuery()</code> will fall through to subsequent
queries. So, if <code class="code">SetFilter()</code> is called before the first query, the same filter
will be in place for the second (and subsequent) queries batched through <code class="code">AddQuery()</code>
unless you call <code class="code">ResetFilters()</code> first. Alternatively, you can add additional filters
as well.</p><p>This would also be true for grouping options and sorting options; no current sorting,
filtering, and grouping settings are affected by this call; so subsequent queries will reuse
current query settings.
</p><p>
<code class="code">AddQuery()</code> returns an index into an array of results
that will be returned from <code class="code">RunQueries()</code> call. It is simply
a sequentially increasing 0-based integer, ie. first call will return 0,
second will return 1, and so on. Just a small helper so you won't have
to track the indexes manualy if you need then.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-runqueries"></a>8.6.3.&nbsp;RunQueries</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function RunQueries ()</p><p>
Connect to searchd, runs a batch of all queries added using <code class="code">AddQuery()</code>,
obtains and returns the result sets. Returns false and sets <code class="code">GetLastError()</code>
message on general error (such as network I/O failure). Returns a plain array
of result sets on success.
</p><p>
Each result set in the returned array is exactly the same as
the result set returned from <a href="#api-func-query" title="8.6.1.&nbsp;Query"><code class="code">Query()</code></a>.
</p><p>
Note that the batch query request itself almost always succeds -
unless there's a network error, blocking index rotation in progress,
or another general failure which prevents the whole request from being
processed.
</p><p>
However individual queries within the batch might very well fail.
In this case their respective result sets will contain non-empty <code class="code">"error"</code> message,
but no matches or query statistics. In the extreme case all queries within the batch
could fail. There still will be no general error reported, because API was able to
succesfully connect to <code class="filename">searchd</code>, submit the batch, and receive
the results - but every result set will have a specific error message.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-resetfilters"></a>8.6.4.&nbsp;ResetFilters</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function ResetFilters ()</p><p>
Clears all currently set filters.
</p><p>
This call is only normally required when using multi-queries. You might want
to set different filters for different queries in the batch. To do that,
you should call <code class="code">ResetFilters()</code> and add new filters using
the respective calls.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-resetgroupby"></a>8.6.5.&nbsp;ResetGroupBy</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function ResetGroupBy ()</p><p>
Clears all currently group-by settings, and disables group-by.
</p><p>
This call is only normally required when using multi-queries.
You can change individual group-by settings using <code class="code">SetGroupBy()</code>
and <code class="code">SetGroupDistinct()</code> calls, but you can not disable
group-by using those calls. <code class="code">ResetGroupBy()</code>
fully resets previous group-by settings and disables group-by mode
in the current state, so that subsequent <code class="code">AddQuery()</code>
calls can perform non-grouping searches.
</p></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-additional-functionality"></a>8.7.&nbsp;Additional functionality</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-buildexcerpts"></a>8.7.1.&nbsp;BuildExcerpts</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function BuildExcerpts ( $docs, $index, $words, $opts=array() )</p><p>
Excerpts (snippets) builder function. Connects to <code class="filename">searchd</code>,
asks it to generate excerpts (snippets) from given documents, and returns the results.
</p><p>
<code class="code">$docs</code> is a plain array of strings that carry the documents' contents.
<code class="code">$index</code> is an index name string. Different settings (such as charset,
morphology, wordforms) from given index will be used.
<code class="code">$words</code> is a string that contains the keywords to highlight. They will
be processed with respect to index settings. For instance, if English stemming
is enabled in the index, "shoes" will be highlighted even if keyword is "shoe".
Starting with version 0.9.9-rc1, keywords can contain wildcards, that work similarly to
<a href="#conf-enable-star" title="11.2.19.&nbsp;enable_star">star-syntax</a> available in queries.
<code class="code">$opts</code> is a hash which contains additional optional highlighting parameters:
</p><div class="variablelist"><dl><dt><span class="term">"before_match":</span></dt>
<dd>A string to insert before a keyword match. Starting with version 1.10-beta,
	a %PASSAGE_ID% macro can be used in this string. The macro is replaced with an incrementing
	passage number within a current snippet. Numbering starts at 1 by default but can be
	overridden with "start_passage_id" option. In a multi-document call, %PASSAGE_ID% would
	restart at every given document. Default is "&lt;b&gt;".</dd><dt><span class="term">"after_match":</span></dt>
<dd>A string to insert after a keyword match. Starting with version 1.10-beta,
	a %PASSAGE_ID% macro can be used in this string. Default is "&lt;b&gt;".</dd><dt><span class="term">"chunk_separator":</span></dt>
<dd>A string to insert between snippet chunks (passages). Default is "&nbsp;...&nbsp;".</dd><dt><span class="term">"limit":</span></dt>
<dd>Maximum snippet size, in symbols (codepoints). Integer, default is 256.</dd><dt><span class="term">"around":</span></dt>
<dd>How much words to pick around each matching keywords block. Integer, default is 5.</dd><dt><span class="term">"exact_phrase":</span></dt>
<dd>Whether to highlight exact query phrase matches only instead of individual keywords. Boolean, default is false.</dd><dt><span class="term">"single_passage":</span></dt>
<dd>Whether to extract single best passage only. Boolean, default is false.</dd><dt><span class="term">"weight_order":</span></dt>
<dd>Whether to sort the extracted passages in order of relevance (decreasing weight),
	or in order of appearance in the document (increasing position). Boolean, default is false.</dd><dt><span class="term">"query_mode":</span></dt>
<dd>Added in version 1.10-beta. Whether to handle $words as a query in
	<a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">extended syntax</a>, or as a bag of words
	(default behavior). For instance, in query mode ("one two" | "three four") will
	only highlight and include those occurrences "one two" or "three four" when 
	the two words from each pair are adjacent to each other. In default mode,
	any single occurrence of "one", "two", "three", or "four" would be
	highlighted. Boolean, default is false.
	</dd><dt><span class="term">"force_all_words":</span></dt>
<dd>Added in version 1.10-beta. Ignores the snippet length limit until it
	includes all the keywords. Boolean, default is false.
	</dd><dt><span class="term">"limit_passages":</span></dt>
<dd>Added in version 1.10-beta. Limits the maximum number of passages
	that can be included into the snippet. Integer, default is 0 (no limit).
	</dd><dt><span class="term">"limit_words":</span></dt>
<dd>Added in version 1.10-beta. Limits the maximum number of keywords
	that can be included into the snippet. Integer, default is 0 (no limit).
	</dd><dt><span class="term">"start_passage_id":</span></dt>
<dd>Added in version 1.10-beta. Specifies the starting value of
	%PASSAGE_ID% macro (that gets detected and expanded in <code class="option">before_match</code>,
	<code class="option">after_match</code> strings). Integer, default is 1.
	</dd><dt><span class="term">"load_files":</span></dt>
<dd>Added in version 1.10-beta. Whether to handle $docs as data
	to extract snippets from (default behavior), or to treat it as file names,
	and load data from specified files on the server side. Boolean, default is false.
	</dd><dt><span class="term">"html_strip_mode":</span></dt>
<dd>Added in version 1.10-beta. HTML stripping mode setting.
	Defaults to "index", which means that index settings will be used.
	The other values are "none" and "strip", that forcibly skip or apply
	stripping irregardless of index settings; and "retain", that retains
	HTML markup and protects it from highlighting. The "retain" mode can
	only be used when highlighting full documents and thus requires that
	no snippet size limits are set. String, allowed values are "none",
	"strip", "index", and "retain".
	</dd><dt><span class="term">"allow_empty":</span></dt>
<dd>Added in version 1.10-beta. Allows empty string to be
	returned as highlighting result when a snippet could not be generated
	(no keywords match, or no passages fit the limit). By default,
	the beginning of original text would be returned instead of an empty
	string. Boolean, default is false.
	</dd></dl></div>
<p>
</p><p>
Returns false on failure. Returns a plain array of strings with excerpts (snippets) on success.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-updateatttributes"></a>8.7.2.&nbsp;UpdateAttributes</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function UpdateAttributes ( $index, $attrs, $values )</p><p>
Instantly updates given attribute values in given documents.
Returns number of actually updated documents (0 or more) on success, or -1 on failure.
</p><p>
<code class="code">$index</code> is a name of the index (or indexes) to be updated.
<code class="code">$attrs</code> is a plain array with string attribute names, listing attributes that are updated.
<code class="code">$values</code> is a hash where key is document ID, and value is a plain array of new attribute values.
</p><p>
<code class="code">$index</code> can be either a single index name or a list, like in <code class="code">Query()</code>.
Unlike <code class="code">Query()</code>, wildcard is not allowed and all the indexes
to update must be specified explicitly. The list of indexes can include
distributed index names. Updates on distributed indexes will be pushed
to all agents.
</p><p>
The updates only work with <code class="code">docinfo=extern</code> storage strategy.
They are very fast because they're working fully in RAM, but they can also
be made persistent: updates are saved on disk on clean <code class="filename">searchd</code>
shutdown initiated by SIGTERM signal. With additional restrictions, updates
are also possible on MVA attributes; refer to <a href="#conf-mva-updates-pool" title="11.4.17.&nbsp;mva_updates_pool">mva_updates_pool</a>
directive for details.
</p><p>
Usage example:
</p><pre class="programlisting">
$cl-&gt;UpdateAttributes ( "test1", array("group_id"), array(1=&gt;array(456)) );
$cl-&gt;UpdateAttributes ( "products", array ( "price", "amount_in_stock" ),
	array ( 1001=&gt;array(123,5), 1002=&gt;array(37,11), 1003=&gt;(25,129) ) );
</pre><p>
The first sample statement will update document 1 in index "test1", setting "group_id" to 456.
The second one will update documents 1001, 1002 and 1003 in index "products". For document 1001,
the new price will be set to 123 and the new amount in stock to 5; for document 1002, the new price
will be 37 and the new amount will be 11; etc.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-buildkeywords"></a>8.7.3.&nbsp;BuildKeywords</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function BuildKeywords ( $query, $index, $hits )</p><p>
Extracts keywords from query using tokenizer settings for given index, optionally with per-keyword occurrence statistics.
Returns an array of hashes with per-keyword information.
</p><p>
<code class="code">$query</code> is a query to extract keywords from.
<code class="code">$index</code> is a name of the index to get tokenizing settings and keyword occurrence statistics from.
<code class="code">$hits</code> is a boolean flag that indicates whether keyword occurrence statistics are required.
</p><p>
Usage example:
</p><pre class="programlisting">
$keywords = $cl-&gt;BuildKeywords ( "this.is.my query", "test1", false );
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-escapestring"></a>8.7.4.&nbsp;EscapeString</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function EscapeString ( $string )</p><p>
Escapes characters that are treated as special operators by the query language parser.
Returns an escaped string.
</p><p>
<code class="code">$string</code> is a string to escape.
</p><p>
This function might seem redundant because it's trivial to implement in any calling
application. However, as the set of special characters might change over time, it makes
sense to have an API call that is guaranteed to escape all such characters at all times.
</p><p>
Usage example:
</p><pre class="programlisting">
$escaped = $cl-&gt;EscapeString ( "escaping-sample@query/string" );
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-status"></a>8.7.5.&nbsp;Status</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function Status ()</p><p>
Queries searchd status, and returns an array of status variable name and value pairs.
</p><p>
Usage example:
</p><pre class="programlisting">
$status = $cl-&gt;Status ();
foreach ( $status as $row )
	print join ( ": ", $row ) . "\n";
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-flushattributes"></a>8.7.6.&nbsp;FlushAttributes</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function FlushAttributes ()</p><p>
Forces <code class="filename">searchd</code> to flush pending attribute updates
to disk, and blocks until completion. Returns a non-negative internal
"flush tag" on success. Returns -1 and sets an error message on error.
Introduced in version 1.10-beta. 
</p><p>
Attribute values updated using <a href="#api-func-updateatttributes" title="8.7.2.&nbsp;UpdateAttributes">UpdateAttributes()</a>
API call are only kept in RAM until a so-called flush (which writes
the current, possibly updated attribute values back to disk). FlushAttributes()
call lets you enforce a flush.  The call will block until <code class="filename">searchd</code>
finishes writing the data to disk, which might take seconds or even minutes
depending on the total data size (.spa file size). All the currently updated
indexes will be flushed.
</p><p>
Flush tag should be treated as an ever growing magic number that does not
mean anything. It's guaranteed to be non-negative. It is guaranteed to grow over
time, though not necessarily in a sequential fashion; for instance, two calls that
return 10 and then 1000 respectively are a valid situation. If two calls to
FlushAttrs() return the same tag, it means that there were no actual attribute
updates in between them, and therefore current flushed state remained the same
(for all indexes).
</p><p>
Usage example:
</p><pre class="programlisting">
$status = $cl-&gt;FlushAttributes ();
if ( $status&lt;0 )
	print "ERROR: " . $cl-&gt;GetLastError(); 
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="api-funcgroup-pconn"></a>8.8.&nbsp;Persistent connections</h2></div></div></div>
<p>
Persistent connections allow to use single network connection to run
multiple commands that would otherwise require reconnects.
</p><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-open"></a>8.8.1.&nbsp;Open</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function Open ()</p><p>
Opens persistent connection to the server.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="api-func-close"></a>8.8.2.&nbsp;Close</h3></div></div></div>
<p><span class="bold"><strong>Prototype:</strong></span> function Close ()</p><p>
Closes previously opened persistent connection.
</p></div></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="sphinxse"></a>Chapter&nbsp;9.&nbsp;MySQL storage engine (SphinxSE)</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#sphinxse-overview">9.1. SphinxSE overview</a></span></dt>
<dt><span class="sect1"><a href="#sphinxse-installing">9.2. Installing SphinxSE</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#sphinxse-mysql50">9.2.1. Compiling MySQL 5.0.x with SphinxSE</a></span></dt>
<dt><span class="sect2"><a href="#sphinxse-mysql51">9.2.2. Compiling MySQL 5.1.x with SphinxSE</a></span></dt>
<dt><span class="sect2"><a href="#sphinxse-checking">9.2.3. Checking SphinxSE installation</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#sphinxse-using">9.3. Using SphinxSE</a></span></dt>
<dt><span class="sect1"><a href="#sphinxse-snippets">9.4. Building snippets (excerpts) via MySQL</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxse-overview"></a>9.1.&nbsp;SphinxSE overview</h2></div></div></div>
<p>
SphinxSE is MySQL storage engine which can be compiled
into MySQL server 5.x using its pluggable architecure.
It is not available for MySQL 4.x series. It also requires
MySQL 5.0.22 or higher in 5.0.x series, or MySQL 5.1.12
or higher in 5.1.x series.
</p><p>
Despite the name, SphinxSE does <span class="emphasis"><em>not</em></span>
actually store any data itself. It is actually a built-in client
which allows MySQL server to talk to <code class="filename">searchd</code>,
run search queries, and obtain search results. All indexing and
searching happen outside MySQL.
</p><p>
Obvious SphinxSE applications include:
</p><div class="itemizedlist"><ul type="disc"><li>easier porting of MySQL FTS applications to Sphinx;</li>
<li>allowing Sphinx use with progamming languages for which native APIs are not available yet;</li>
<li>optimizations when additional Sphinx result set processing on MySQL side is required
	(eg. JOINs with original document tables, additional MySQL-side filtering, etc).</li>
</ul></div>
<p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxse-installing"></a>9.2.&nbsp;Installing SphinxSE</h2></div></div></div>
<p>
You will need to obtain a copy of MySQL sources, prepare those,
and then recompile MySQL binary.
MySQL sources (mysql-5.x.yy.tar.gz) could be obtained from 
<a href="http://dev.mysql.com" target="_top">dev.mysql.com</a> Web site.
</p><p>
For some MySQL versions, there are delta tarballs with already
prepared source versions available from Sphinx Web site. After unzipping
those over original sources MySQL would be ready to be configured and
built with Sphinx support.
</p><p>
If such tarball is not available, or does not work for you for any
reason, you would have to prepare sources manually. You will need to
GNU Autotools framework (autoconf, automake and libtool) installed
to do that.
</p><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="sphinxse-mysql50"></a>9.2.1.&nbsp;Compiling MySQL 5.0.x with SphinxSE</h3></div></div></div>
<p>
Skips steps 1-3 if using already prepared delta tarball.
</p><div class="orderedlist"><ol type="1"><li><p>copy <code class="filename">sphinx.5.0.yy.diff</code> patch file
into MySQL sources directory and run
</p><pre class="programlisting">
patch -p1 &lt; sphinx.5.0.yy.diff
</pre><p>
If there's no .diff file exactly for the specific version you need
to build, try applying .diff with closest version numbers. It is important
that the patch should apply with no rejects.
</p></li>
<li>in MySQL sources directory, run
<pre class="programlisting">
sh BUILD/autorun.sh
</pre></li>
<li>in MySQL sources directory, create <code class="filename">sql/sphinx</code>
directory in and copy all files in <code class="filename">mysqlse</code> directory 
from Sphinx sources there. Example:
<pre class="programlisting">
cp -R /root/builds/sphinx-0.9.7/mysqlse /root/builds/mysql-5.0.24/sql/sphinx
</pre></li>
<li>
configure MySQL and enable Sphinx engine:
<pre class="programlisting">
./configure --with-sphinx-storage-engine
</pre></li>
<li>
build and install MySQL:
<pre class="programlisting">
make
make install
</pre></li>
</ol></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="sphinxse-mysql51"></a>9.2.2.&nbsp;Compiling MySQL 5.1.x with SphinxSE</h3></div></div></div>
<p>
Skip steps 1-2 if using already prepared delta tarball.
</p><div class="orderedlist"><ol type="1"><li>in MySQL sources directory, create <code class="filename">storage/sphinx</code>
directory in and copy all files in <code class="filename">mysqlse</code> directory 
from Sphinx sources there. Example:
<pre class="programlisting">
cp -R /root/builds/sphinx-0.9.7/mysqlse /root/builds/mysql-5.1.14/storage/sphinx
</pre></li>
<li>in MySQL sources directory, run
<pre class="programlisting">
sh BUILD/autorun.sh
</pre></li>
<li>
configure MySQL and enable Sphinx engine:
<pre class="programlisting">
./configure --with-plugins=sphinx
</pre></li>
<li>
build and install MySQL:
<pre class="programlisting">
make
make install
</pre></li>
</ol></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="sphinxse-checking"></a>9.2.3.&nbsp;Checking SphinxSE installation</h3></div></div></div>

To check whether SphinxSE has been succesfully compiled
into MySQL, launch newly built servers, run mysql client and
issue <code class="code">SHOW ENGINES</code> query. You should see a list
of all available engines. Sphinx should be present and "Support"
column should contain "YES":

<pre class="programlisting">
mysql&gt; show engines;
+------------+----------+-------------------------------------------------------------+
| Engine     | Support  | Comment                                                     |
+------------+----------+-------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance      |
  ...
| SPHINX     | YES      | Sphinx storage engine                                       |
  ...
+------------+----------+-------------------------------------------------------------+
13 rows in set (0.00 sec)
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxse-using"></a>9.3.&nbsp;Using SphinxSE</h2></div></div></div>
<p>
To search via SphinxSE, you would need to create special ENGINE=SPHINX "search table",
and then SELECT from it with full text query put into WHERE clause for query column.
</p><p>
Let's begin with an example create statement and search query:
</p><pre class="programlisting">
CREATE TABLE t1
(
    id          INTEGER UNSIGNED NOT NULL,
    weight      INTEGER NOT NULL,
    query       VARCHAR(3072) NOT NULL,
    group_id    INTEGER,
    INDEX(query)
) ENGINE=SPHINX CONNECTION="sphinx://localhost:9312/test";

SELECT * FROM t1 WHERE query='test it;mode=any';
</pre><p>
</p><p>
First 3 columns of search table <span class="emphasis"><em>must</em></span> have a types of
<code class="code">INTEGER UNSINGED</code> or <code class="code">BIGINT</code> for the 1st column (document id),
<code class="code">INTEGER</code> or <code class="code">BIGINT</code> for the 2nd column (match weight), and
<code class="code">VARCHAR</code> or <code class="code">TEXT</code> for the 3rd column (your query), respectively.
This mapping is fixed; you can not omit any of these three required columns,
or move them around, or change types. Also, query column must be indexed;
all the others must be kept unindexed. Columns' names are ignored so you
can use arbitrary ones.
</p><p>
Additional columns must be either <code class="code">INTEGER</code>, <code class="code">TIMESTAMP</code>,
<code class="code">BIGINT</code>, <code class="code">VARCHAR</code>, or <code class="code">FLOAT</code>.
They will be bound to attributes provided in Sphinx result set by name, so their
names must match attribute names specified in <code class="filename">sphinx.conf</code>.
If there's no such attribute name in Sphinx search results, column will have
<code class="code">NULL</code> values.
</p><p>
Special "virtual" attributes names can also be bound to SphinxSE columns.
<code class="code">_sph_</code> needs to be used instead of <code class="code">@</code> for that.
For instance, to obtain the values of <code class="code">@groupby</code>, <code class="code">@count</code>,
or <code class="code">@distinct</code> virtual attributes, use <code class="code">_sph_groupby</code>,
<code class="code">_sph_count</code> or <code class="code">_sph_distinct</code> column names, respectively.
</p><p>
<code class="code">CONNECTION</code> string parameter can be used to specify default
searchd host, port and indexes for queries issued using this table.
If no connection string is specified in <code class="code">CREATE TABLE</code>,
index name "*" (ie. search all indexes) and localhost:9312 are assumed.
Connection string syntax is as follows:
</p><pre class="programlisting">
CONNECTION="sphinx://HOST:PORT/INDEXNAME"
</pre><p>
You can change the default connection string later:
</p><pre class="programlisting">
ALTER TABLE t1 CONNECTION="sphinx://NEWHOST:NEWPORT/NEWINDEXNAME";
</pre><p>
You can also override all these parameters per-query.
</p><p>
As seen in example, both query text and search options should be put
into WHERE clause on search query column (ie. 3rd column); the options
are separated by semicolons; and their names from values by equality sign.
Any number of options can be specified. Available options are:
</p><div class="itemizedlist"><ul type="disc"><li>query - query text;</li>
<li>mode - matching mode. Must be one of "all", "any", "phrase",
	"boolean", or "extended". Default is "all";</li>
<li>sort - match sorting mode. Must be one of "relevance", "attr_desc",
"attr_asc", "time_segments", or "extended". In all modes besides "relevance"
attribute name (or sorting clause for "extended") is also required after a colon:
<pre class="programlisting">
... WHERE query='test;sort=attr_asc:group_id';
... WHERE query='test;sort=extended:@weight desc, group_id asc';
</pre></li>
<li>offset - offset into result set, default is 0;</li>
<li>limit - amount of matches to retrieve from result set, default is 20;</li>
<li>index - names of the indexes to search:
<pre class="programlisting">
... WHERE query='test;index=test1;';
... WHERE query='test;index=test1,test2,test3;';
</pre></li>
<li>minid, maxid - min and max document ID to match;</li>
<li>weights - comma-separated list of weights to be assigned to Sphinx full-text fields:
<pre class="programlisting">
... WHERE query='test;weights=1,2,3;';
</pre></li>
<li>filter, !filter - comma-separated attribute name and a set of values to match:
<pre class="programlisting">
# only include groups 1, 5 and 19
... WHERE query='test;filter=group_id,1,5,19;';

# exclude groups 3 and 11
... WHERE query='test;!filter=group_id,3,11;';
</pre></li>
<li>range, !range - comma-separated attribute name, min and max value to match:
<pre class="programlisting">
# include groups from 3 to 7, inclusive
... WHERE query='test;range=group_id,3,7;';

# exclude groups from 5 to 25
... WHERE query='test;!range=group_id,5,25;';
</pre></li>
<li>maxmatches - per-query max matches value:
<pre class="programlisting">
... WHERE query='test;maxmatches=2000;';
</pre></li>
<li>groupby - group-by function and attribute:
<pre class="programlisting">
... WHERE query='test;groupby=day:published_ts;';
... WHERE query='test;groupby=attr:group_id;';
</pre></li>
<li>groupsort - group-by sorting clause:
<pre class="programlisting">
... WHERE query='test;groupsort=@count desc;';
</pre></li>
<li>indexweights - comma-separated list of index names and weights
to use when searching through several indexes:
<pre class="programlisting">
... WHERE query='test;indexweights=idx_exact,2,idx_stemmed,1;';
</pre></li>
<li>comment - a string to mark this query in query log
(mapping to $comment parameter in <a href="#api-func-query" title="8.6.1.&nbsp;Query">Query()</a> API call):
<pre class="programlisting">
... WHERE query='test;comment=marker001;';
</pre></li>
<li>select - a string with expressions to compute
(mapping to <a href="#api-func-setselect" title="8.2.4.&nbsp;SetSelect">SetSelect()</a> API call):
<pre class="programlisting">
... WHERE query='test;select=2*a+3*b as myexpr;';
</pre></li>
</ul></div>
<p>
</p><p>
One <span class="bold"><strong>very important</strong></span> note that it is
<span class="bold"><strong>much</strong></span> more efficient to allow Sphinx
to perform sorting, filtering and slicing the result set than to raise
max matches count and use WHERE, ORDER BY and LIMIT clauses on MySQL
side. This is for two reasons. First, Sphinx does a number of
optimizations and performs better than MySQL on these tasks.
Second, less data would need to be packed by searchd, transferred
and unpacked by SphinxSE.
</p><p>
Starting with version 0.9.9-rc1, additional query info besides result set could be
retrieved with <code class="code">SHOW ENGINE SPHINX STATUS</code> statement:
</p><pre class="programlisting">
mysql&gt; SHOW ENGINE SPHINX STATUS;
+--------+-------+-------------------------------------------------+
| Type   | Name  | Status                                          |
+--------+-------+-------------------------------------------------+
| SPHINX | stats | total: 25, total found: 25, time: 126, words: 2 | 
| SPHINX | words | sphinx:591:1256 soft:11076:15945                | 
+--------+-------+-------------------------------------------------+
2 rows in set (0.00 sec)
</pre><p>
This information can also be accessed through status variables. Note
that this method does not require super-user privileges.
</p><pre class="programlisting">
mysql&gt; SHOW STATUS LIKE 'sphinx_%';
+--------------------+----------------------------------+
| Variable_name      | Value                            |
+--------------------+----------------------------------+
| sphinx_total       | 25                               | 
| sphinx_total_found | 25                               | 
| sphinx_time        | 126                              | 
| sphinx_word_count  | 2                                | 
| sphinx_words       | sphinx:591:1256 soft:11076:15945 | 
+--------------------+----------------------------------+
5 rows in set (0.00 sec)
</pre><p>
</p><p>
You could perform JOINs on SphinxSE search table and tables using
other engines. Here's an example with "documents" from example.sql:
</p><pre class="programlisting">
mysql&gt; SELECT content, date_added FROM test.documents docs
-&gt; JOIN t1 ON (docs.id=t1.id) 
-&gt; WHERE query="one document;mode=any";
+-------------------------------------+---------------------+
| content                             | docdate             |
+-------------------------------------+---------------------+
| this is my test document number two | 2006-06-17 14:04:28 | 
| this is my test document number one | 2006-06-17 14:04:28 | 
+-------------------------------------+---------------------+
2 rows in set (0.00 sec)

mysql&gt; SHOW ENGINE SPHINX STATUS;
+--------+-------+---------------------------------------------+
| Type   | Name  | Status                                      |
+--------+-------+---------------------------------------------+
| SPHINX | stats | total: 2, total found: 2, time: 0, words: 2 | 
| SPHINX | words | one:1:2 document:2:2                        | 
+--------+-------+---------------------------------------------+
2 rows in set (0.00 sec)
</pre><p>
</p></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sphinxse-snippets"></a>9.4.&nbsp;Building snippets (excerpts) via MySQL</h2></div></div></div>
<p>
Starting with version 0.9.9-rc2, SphinxSE also includes a UDF function
that lets you create snippets through MySQL. The functionality is fully
similar to <a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">BuildExcerprts</a>
API call but accesible through MySQL+SphinxSE.
</p><p>
The binary that provides the UDF is named <code class="filename">sphinx.so</code>
and should be automatically built and installed to proper location
along with SphinxSE itself. If it does not get installed automatically
for some reason, look for <code class="filename">sphinx.so</code> in the build
directory and copy it to the plugins directory of your MySQL instance.
After that, register the UDF using the following statement:
</p><pre class="programlisting">
CREATE FUNCTION sphinx_snippets RETURNS STRING SONAME 'sphinx.so';
</pre><p>
</p><p>
Function name <span class="emphasis"><em>must</em></span> be sphinx_snippets,
you can not use an arbitrary name. Function arguments are as follows:
</p><p>
<span class="bold"><strong>Prototype:</strong></span> function sphinx_snippets ( document, index, words, [options] );
</p><p>
Document and words arguments can be either strings or table columns.
Options must be specified like this: <code class="code">'value' AS option_name</code>.
For a list of supported options, refer to
<a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">BuildExcerprts()</a> API call.
The only UDF-specific additional option is named <code class="code">'sphinx'</code>
and lets you specify searchd location (host and port).
</p><p>
Usage examples:
</p><pre class="programlisting">
SELECT sphinx_snippets('hello world doc', 'main', 'world',
    'sphinx://192.168.1.1/' AS sphinx, true AS exact_phrase,
    '[b]' AS before_match, '[/b]' AS after_match)
FROM documents;

SELECT title, sphinx_snippets(text, 'index', 'mysql php') AS text
    FROM sphinx, documents
    WHERE query='mysql php' AND sphinx.id=documents.id;
</pre><p>
</p></div></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="reporting-bugs"></a>Chapter&nbsp;10.&nbsp;Reporting bugs</h2></div></div></div>
<p>
Unfortunately, Sphinx is not yet 100% bug free (even though I'm working hard
towards that), so you might occasionally run into some issues.
</p><p>
Reporting as much as possible about each bug is very important -
because to fix it, I need to be able either to reproduce and debug the bug,
or to deduce what's causing it from the information that you provide.
So here are some instructions on how to do that.
</p><h2>Build-time issues</h2><p>If Sphinx fails to build for some reason, please do the following:</p><div class="orderedlist"><ol type="1"><li>check that headers and libraries for your DBMS are properly installed
(for instance, check that <code class="filename">mysql-devel</code> package is present);
</li>
<li>report Sphinx version and config file (be sure to remove the passwords!),
MySQL (or PostgreSQL) configuration info, gcc version, OS version and CPU type
(ie. x86, x86-64, PowerPC, etc):
<pre class="programlisting">
mysql_config
gcc --version
uname -a
</pre></li>
<li>
report the error message which is produced by <code class="filename">configure</code>
or <code class="filename">gcc</code> (it should be to include error message itself only,
not the whole build log).
</li>
</ol></div>
<h2>Run-time issues</h2><p>
If Sphinx builds and runs, but there are any problems running it,
please do the following:
</p><div class="orderedlist"><ol type="1"><li>describe the bug (ie. both the expected behavior and actual behavior)
and all the steps necessary to reproduce it;</li>
<li>include Sphinx version and config file (be sure to remove the passwords!),
MySQL (or PostgreSQL) version, gcc version, OS version and CPU type (ie. x86, x86-64,
PowerPC, etc):
<pre class="programlisting">
mysql --version
gcc --version
uname -a
</pre></li>
<li>build, install and run debug versions of all Sphinx programs (this is
to enable a lot of additional internal checks, so-called assertions):
<pre class="programlisting">
make distclean
./configure --with-debug
make install
killall -TERM searchd
</pre></li>
<li>reindex to check if any assertions are triggered (in this case,
it's likely that the index is corrupted and causing problems);
</li>
<li>if the bug does not reproduce with debug versions,
revert to non-debug and mention it in your report;
</li>
<li>if the bug could be easily reproduced with a small (1-100 record)
part of your database, please provide a gzipped dump of that part;
</li>
<li>if the problem is related to <code class="filename">searchd</code>, include
relevant entries from <code class="filename">searchd.log</code> and
<code class="filename">query.log</code> in your bug report;
</li>
<li>if the problem is related to <code class="filename">searchd</code>, try
running it in console mode and check if it dies with an assertion:
<pre class="programlisting">
./searchd --console
</pre></li>
<li>if any program dies with an assertion, provide the assertion message.</li>
</ol></div>
<h2>Debugging assertions, crashes and hangups</h2><p>
If any program dies with an assertion, crashes without an assertion or hangs up,
you would additionally need to generate a core dump and examine it.
</p><div class="orderedlist"><ol type="1"><li>
enable core dumps. On most Linux systems, this is done
using <code class="filename">ulimit</code>:
<pre class="programlisting">
ulimit -c 32768
</pre></li>
<li>
run the program and try to reproduce the bug;
</li>
<li>
if the program crashes (either with or without an assertion),
find the core file in current directory (it should typically print
out "Segmentation fault (core dumped)" message);
</li>
<li>
if the program hangs, use <code class="filename">kill -SEGV</code>
from another console to force it to exit and dump core:
<pre class="programlisting">
kill -SEGV HANGED-PROCESS-ID
</pre></li>
<li>
use <code class="filename">gdb</code> to examine the core file
and obtain a backtrace:
<pre class="programlisting">
gdb ./CRASHED-PROGRAM-FILE-NAME CORE-DUMP-FILE-NAME
(gdb) bt
(gdb) quit
</pre></li>
</ol></div>
<p>
Note that HANGED-PROCESS-ID, CRASHED-PROGRAM-FILE-NAME and
CORE-DUMP-FILE-NAME must all be replaced with specific numbers
and file names. For example, hanged searchd debugging session
would look like:
</p><pre class="programlisting">
# kill -SEGV 12345
# ls *core*
core.12345
# gdb ./searchd core.12345
(gdb) bt
...
(gdb) quit
</pre><p>
</p><p>
Note that <code class="filename">ulimit</code> is not server-wide
and only affects current shell session. This means that you will not
have to restore any server-wide limits - but if you relogin,
you will have to set <code class="filename">ulimit</code> again.
</p><p>
Core dumps should be placed in current working directory
(and Sphinx programs do not change it), so this is where you
would look for them.
</p><p>
Please do not immediately remove the core file because there could
be additional helpful information which could be retrieved from it.
You do not need to send me this file (as the debug info there is
closely tied to your system) but I might need to ask
you a few additional questions about it.
</p></div>
<div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="conf-reference"></a>Chapter&nbsp;11.&nbsp;<code class="filename">sphinx.conf</code> options reference</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#confgroup-source">11.1. Data source configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-source-type">11.1.1. type</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-host">11.1.2. sql_host</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-port">11.1.3. sql_port</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-user">11.1.4. sql_user</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-pass">11.1.5. sql_pass</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-db">11.1.6. sql_db</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-sock">11.1.7. sql_sock</a></span></dt>
<dt><span class="sect2"><a href="#conf-mysql-connect-flags">11.1.8. mysql_connect_flags</a></span></dt>
<dt><span class="sect2"><a href="#conf-mysql-ssl">11.1.9. mysql_ssl_cert, mysql_ssl_key, mysql_ssl_ca</a></span></dt>
<dt><span class="sect2"><a href="#conf-odbc-dsn">11.1.10. odbc_dsn</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-pre">11.1.11. sql_query_pre</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query">11.1.12. sql_query</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-joined-field">11.1.13. sql_joined_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-range">11.1.14. sql_query_range</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-range-step">11.1.15. sql_range_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-killlist">11.1.16. sql_query_killlist</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-uint">11.1.17. sql_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-bool">11.1.18. sql_attr_bool</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-bigint">11.1.19. sql_attr_bigint</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-timestamp">11.1.20. sql_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-str2ordinal">11.1.21. sql_attr_str2ordinal</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-float">11.1.22. sql_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-multi">11.1.23. sql_attr_multi</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-string">11.1.24. sql_attr_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-attr-str2wordcount">11.1.25. sql_attr_str2wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-field-string">11.1.26. sql_field_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-field-str2wordcount">11.1.27. sql_field_str2wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-file-field">11.1.28. sql_file_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-post">11.1.29. sql_query_post</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-post-index">11.1.30. sql_query_post_index</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-ranged-throttle">11.1.31. sql_ranged_throttle</a></span></dt>
<dt><span class="sect2"><a href="#conf-sql-query-info">11.1.32. sql_query_info</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-command">11.1.33. xmlpipe_command</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field">11.1.34. xmlpipe_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field-string">11.1.35. xmlpipe_field_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-field-wordcount">11.1.36. xmlpipe_field_wordcount</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-uint">11.1.37. xmlpipe_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-bool">11.1.38. xmlpipe_attr_bool</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-timestamp">11.1.39. xmlpipe_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-str2ordinal">11.1.40. xmlpipe_attr_str2ordinal</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-float">11.1.41. xmlpipe_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-multi">11.1.42. xmlpipe_attr_multi</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-attr-string">11.1.43. xmlpipe_attr_string</a></span></dt>
<dt><span class="sect2"><a href="#conf-xmlpipe-fixup-utf8">11.1.44. xmlpipe_fixup_utf8</a></span></dt>
<dt><span class="sect2"><a href="#conf-mssql-winauth">11.1.45. mssql_winauth</a></span></dt>
<dt><span class="sect2"><a href="#conf-mssql-unicode">11.1.46. mssql_unicode</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-zlib">11.1.47. unpack_zlib</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-mysqlcompress">11.1.48. unpack_mysqlcompress</a></span></dt>
<dt><span class="sect2"><a href="#conf-unpack-mysqlcompress-maxsize">11.1.49. unpack_mysqlcompress_maxsize</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-index">11.2. Index configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-index-type">11.2.1. type</a></span></dt>
<dt><span class="sect2"><a href="#conf-source">11.2.2. source</a></span></dt>
<dt><span class="sect2"><a href="#conf-path">11.2.3. path</a></span></dt>
<dt><span class="sect2"><a href="#conf-docinfo">11.2.4. docinfo</a></span></dt>
<dt><span class="sect2"><a href="#conf-mlock">11.2.5. mlock</a></span></dt>
<dt><span class="sect2"><a href="#conf-morphology">11.2.6. morphology</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-stemming-len">11.2.7. min_stemming_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-stopwords">11.2.8. stopwords</a></span></dt>
<dt><span class="sect2"><a href="#conf-wordforms">11.2.9. wordforms</a></span></dt>
<dt><span class="sect2"><a href="#conf-exceptions">11.2.10. exceptions</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-word-len">11.2.11. min_word_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-charset-type">11.2.12. charset_type</a></span></dt>
<dt><span class="sect2"><a href="#conf-charset-table">11.2.13. charset_table</a></span></dt>
<dt><span class="sect2"><a href="#conf-ignore-chars">11.2.14. ignore_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-prefix-len">11.2.15. min_prefix_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-min-infix-len">11.2.16. min_infix_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-prefix-fields">11.2.17. prefix_fields</a></span></dt>
<dt><span class="sect2"><a href="#conf-infix-fields">11.2.18. infix_fields</a></span></dt>
<dt><span class="sect2"><a href="#conf-enable-star">11.2.19. enable_star</a></span></dt>
<dt><span class="sect2"><a href="#conf-ngram-len">11.2.20. ngram_len</a></span></dt>
<dt><span class="sect2"><a href="#conf-ngram-chars">11.2.21. ngram_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-phrase-boundary">11.2.22. phrase_boundary</a></span></dt>
<dt><span class="sect2"><a href="#conf-phrase-boundary-step">11.2.23. phrase_boundary_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-strip">11.2.24. html_strip</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-index-attrs">11.2.25. html_index_attrs</a></span></dt>
<dt><span class="sect2"><a href="#conf-html-remove-elements">11.2.26. html_remove_elements</a></span></dt>
<dt><span class="sect2"><a href="#conf-local">11.2.27. local</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent">11.2.28. agent</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-blackhole">11.2.29. agent_blackhole</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-connect-timeout">11.2.30. agent_connect_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-agent-query-timeout">11.2.31. agent_query_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-preopen">11.2.32. preopen</a></span></dt>
<dt><span class="sect2"><a href="#conf-ondisk-dict">11.2.33. ondisk_dict</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-enable">11.2.34. inplace_enable</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-hit-gap">11.2.35. inplace_hit_gap</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-docinfo-gap">11.2.36. inplace_docinfo_gap</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-reloc-factor">11.2.37. inplace_reloc_factor</a></span></dt>
<dt><span class="sect2"><a href="#conf-inplace-write-factor">11.2.38. inplace_write_factor</a></span></dt>
<dt><span class="sect2"><a href="#conf-index-exact-words">11.2.39. index_exact_words</a></span></dt>
<dt><span class="sect2"><a href="#conf-overshort-step">11.2.40. overshort_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-stopword-step">11.2.41. stopword_step</a></span></dt>
<dt><span class="sect2"><a href="#conf-hitless-words">11.2.42. hitless_words</a></span></dt>
<dt><span class="sect2"><a href="#conf-expand-keywords">11.2.43. expand_keywords</a></span></dt>
<dt><span class="sect2"><a href="#conf-blend-chars">11.2.44. blend_chars</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-mem-limit">11.2.45. rt_mem_limit</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-field">11.2.46. rt_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-uint">11.2.47. rt_attr_uint</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-bigint">11.2.48. rt_attr_bigint</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-float">11.2.49. rt_attr_float</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-timestamp">11.2.50. rt_attr_timestamp</a></span></dt>
<dt><span class="sect2"><a href="#conf-rt-attr-string">11.2.51. rt_attr_string</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-indexer">11.3. <code class="filename">indexer</code> program configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-mem-limit">11.3.1. mem_limit</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-iops">11.3.2. max_iops</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-iosize">11.3.3. max_iosize</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-xmlpipe2-field">11.3.4. max_xmlpipe2_field</a></span></dt>
<dt><span class="sect2"><a href="#conf-write-buffer">11.3.5. write_buffer</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-file-field-buffer">11.3.6. max_file_field_buffer</a></span></dt>
</dl></dd><dt><span class="sect1"><a href="#confgroup-searchd">11.4. <code class="filename">searchd</code> program configuration options</a></span></dt>
<dd><dl><dt><span class="sect2"><a href="#conf-listen">11.4.1. listen</a></span></dt>
<dt><span class="sect2"><a href="#conf-address">11.4.2. address</a></span></dt>
<dt><span class="sect2"><a href="#conf-port">11.4.3. port</a></span></dt>
<dt><span class="sect2"><a href="#conf-log">11.4.4. log</a></span></dt>
<dt><span class="sect2"><a href="#conf-query-log">11.4.5. query_log</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-timeout">11.4.6. read_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-client-timeout">11.4.7. client_timeout</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-children">11.4.8. max_children</a></span></dt>
<dt><span class="sect2"><a href="#conf-pid-file">11.4.9. pid_file</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-matches">11.4.10. max_matches</a></span></dt>
<dt><span class="sect2"><a href="#conf-seamless-rotate">11.4.11. seamless_rotate</a></span></dt>
<dt><span class="sect2"><a href="#conf-preopen-indexes">11.4.12. preopen_indexes</a></span></dt>
<dt><span class="sect2"><a href="#conf-unlink-old">11.4.13. unlink_old</a></span></dt>
<dt><span class="sect2"><a href="#conf-attr-flush-period">11.4.14. attr_flush_period</a></span></dt>
<dt><span class="sect2"><a href="#conf-ondisk-dict-default">11.4.15. ondisk_dict_default</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-packet-size">11.4.16. max_packet_size</a></span></dt>
<dt><span class="sect2"><a href="#conf-mva-updates-pool">11.4.17. mva_updates_pool</a></span></dt>
<dt><span class="sect2"><a href="#conf-crash-log-path">11.4.18. crash_log_path</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-filters">11.4.19. max_filters</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-filter-values">11.4.20. max_filter_values</a></span></dt>
<dt><span class="sect2"><a href="#conf-listen-backlog">11.4.21. listen_backlog</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-buffer">11.4.22. read_buffer</a></span></dt>
<dt><span class="sect2"><a href="#conf-read-unhinted">11.4.23. read_unhinted</a></span></dt>
<dt><span class="sect2"><a href="#conf-max-batch-queries">11.4.24. max_batch_queries</a></span></dt>
<dt><span class="sect2"><a href="#conf-subtree-docs-cache">11.4.25. subtree_docs_cache</a></span></dt>
<dt><span class="sect2"><a href="#conf-subtree-hits-cache">11.4.26. subtree_hits_cache</a></span></dt>
<dt><span class="sect2"><a href="#conf-workers">11.4.27. workers</a></span></dt>
<dt><span class="sect2"><a href="#conf-dist-threads">11.4.28. dist_threads</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-path">11.4.29. binlog_path</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-flush">11.4.30. binlog_flush</a></span></dt>
<dt><span class="sect2"><a href="#conf-binlog-max-log-size">11.4.31. binlog_max_log_size</a></span></dt>
</dl></dd></dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="confgroup-source"></a>11.1.&nbsp;Data source configuration options</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-source-type"></a>11.1.1.&nbsp;type</h3></div></div></div>
<p>
Data source type.
Mandatory, no default value.
Known types are <code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>,
<code class="option">xmlpipe</code> and <code class="option">xmlpipe2</code>, and <code class="option">odbc</code>.
</p><p>
All other per-source options depend on source type selected by this option.
Names of the options used for SQL sources (ie. MySQL, PostgreSQL, MS SQL) start with "sql_";
names of the ones used for xmlpipe and xmlpipe2 start with "xmlpipe_".
All source types except <code class="option">xmlpipe</code> are conditional; they might or might
not be supported depending on your build settings, installed client libraries, etc.
<code class="option">mssql</code> type is currently only available on Windows.
<code class="option">odbc</code> type is available both on Windows natively and on
Linux through <a href="http://www.unixodbc.org/" target="_top">UnixODBC library</a>.
</p><h4>Example:</h4><pre class="programlisting">
type = mysql
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-host"></a>11.1.2.&nbsp;sql_host</h3></div></div></div>
<p>
SQL server host to connect to.
Mandatory, no default value.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
In the simplest case when Sphinx resides on the same host with your MySQL
or PostgreSQL installation, you would simply specify "localhost". Note that
MySQL client library chooses whether to connect over TCP/IP or over UNIX
socket based on the host name. Specifically "localhost" will force it
to use UNIX socket (this is the default and generally recommended mode)
and "127.0.0.1" will force TCP/IP usage. Refer to
<a href="http://dev.mysql.com/doc/refman/5.0/en/mysql-real-connect.html" target="_top">MySQL manual</a>
for more details.
</p><h4>Example:</h4><pre class="programlisting">
sql_host = localhost
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-port"></a>11.1.3.&nbsp;sql_port</h3></div></div></div>
<p>
SQL server IP port to connect to.
Optional, default is 3306 for <code class="option">mysql</code> source type and 5432 for <code class="option">pgsql</code> type.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Note that it depends on <a href="#conf-sql-host" title="11.1.2.&nbsp;sql_host">sql_host</a> setting whether this value will actually be used.
</p><h4>Example:</h4><pre class="programlisting">
sql_port = 3306
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-user"></a>11.1.4.&nbsp;sql_user</h3></div></div></div>
<p>
SQL user to use when connecting to <a href="#conf-sql-host" title="11.1.2.&nbsp;sql_host">sql_host</a>.
Mandatory, no default value.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><h4>Example:</h4><pre class="programlisting">
sql_user = test
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-pass"></a>11.1.5.&nbsp;sql_pass</h3></div></div></div>
<p>
SQL user password to use when connecting to <a href="#conf-sql-host" title="11.1.2.&nbsp;sql_host">sql_host</a>.
Mandatory, no default value.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><h4>Example:</h4><pre class="programlisting">
sql_pass = mysecretpassword
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-db"></a>11.1.6.&nbsp;sql_db</h3></div></div></div>
<p>
SQL database (in MySQL terms) to use after the connection and perform further queries within.
Mandatory, no default value.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><h4>Example:</h4><pre class="programlisting">
sql_db = test
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-sock"></a>11.1.7.&nbsp;sql_sock</h3></div></div></div>
<p>
UNIX socket name to connect to for local SQL servers.
Optional, default value is empty (use client library default settings).
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
On Linux, it would typically be <code class="filename">/var/lib/mysql/mysql.sock</code>.
On FreeBSD, it would typically be <code class="filename">/tmp/mysql.sock</code>.
Note that it depends on <a href="#conf-sql-host" title="11.1.2.&nbsp;sql_host">sql_host</a> setting whether this value will actually be used.
</p><h4>Example:</h4><pre class="programlisting">
sql_sock = /tmp/mysql.sock
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mysql-connect-flags"></a>11.1.8.&nbsp;mysql_connect_flags</h3></div></div></div>
<p>
MySQL client connection flags.
Optional, default value is 0 (do not set any flags).
Applies to <code class="option">mysql</code> source type only.
</p><p>
This option must contain an integer value with the sum of the flags.
The value will be passed to <a href="http://dev.mysql.com/doc/refman/5.0/en/mysql-real-connect.html" target="_top">mysql_real_connect()</a> verbatim.
The flags are enumerated in mysql_com.h include file.
Flags that are especially interesting in regard to indexing, with their respective values, are as follows:
</p><div class="itemizedlist"><ul type="disc"><li>CLIENT_COMPRESS = 32; can use compression protocol</li>
<li>CLIENT_SSL = 2048; switch to SSL after handshake</li>
<li>CLIENT_SECURE_CONNECTION = 32768; new 4.1 authentication</li>
</ul></div>
<p>
For instance, you can specify 2080 (2048+32) to use both compression and SSL,
or 32768 to use new authentication only. Initially, this option was introduced
to be able to use compression when the <code class="filename">indexer</code>
and <code class="filename">mysqld</code> are on different hosts. Compression on 1 Gbps
links is most likely to hurt indexing time though it reduces network traffic,
both in theory and in practice. However, enabling compression on 100 Mbps links
may improve indexing time significantly (upto 20-30% of the total indexing time
improvement was reported). Your mileage may vary.
</p><h4>Example:</h4><pre class="programlisting">
mysql_connect_flags = 32 # enable compression
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mysql-ssl"></a>11.1.9.&nbsp;mysql_ssl_cert, mysql_ssl_key, mysql_ssl_ca</h3></div></div></div>
<p>
SSL certificate settings to use for connecting to MySQL server.
Optional, default values are empty strings (do not use SSL).
Applies to <code class="option">mysql</code> source type only.
</p><p>
These directives let you set up secure SSL connection between
<code class="filename">indexer</code> and MySQL. The details on creating
the certificates and setting up MySQL server can be found in
MySQL documentation.
</p><h4>Example:</h4><pre class="programlisting">
mysql_ssl_cert = /etc/ssl/client-cert.pem
mysql_ssl_key = /etc/ssl/client-key.pem
mysql_ssl_ca = /etc/ssl/cacert.pem
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-odbc-dsn"></a>11.1.10.&nbsp;odbc_dsn</h3></div></div></div>
<p>
ODBC DSN to connect to.
Mandatory, no default value.
Applies to <code class="option">odbc</code> source type only.
</p><p>
ODBC DSN (Data Source Name) specifies the credentials (host, user, password, etc)
to use when connecting to ODBC data source. The format depends on specific ODBC
driver used.
</p><h4>Example:</h4><pre class="programlisting">
odbc_dsn = Driver={Oracle ODBC Driver};Dbq=myDBName;Uid=myUsername;Pwd=myPassword
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-pre"></a>11.1.11.&nbsp;sql_query_pre</h3></div></div></div>
<p>
Pre-fetch query, or pre-query.
Multi-value, optional, default is empty list of queries.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
Multi-value means that you can specify several pre-queries.
They are executed before <a href="#conf-sql-query" title="11.1.12.&nbsp;sql_query">the main fetch query</a>,
and they will be exectued exactly in order of appeareance in the configuration file.
Pre-query results are ignored.
</p><p>
Pre-queries are useful in a lot of ways. They are used to setup encoding,
mark records that are going to be indexed, update internal counters,
set various per-connection SQL server options and variables, and so on.
</p><p>
Perhaps the most frequent pre-query usage is to specify the encoding
that the server will use for the rows it returnes. It <span class="bold"><strong>must</strong></span> match
the encoding that Sphinx expects (as specified by <a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a>
and <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> options).
Two MySQL specific examples of setting the encoding are:
</p><pre class="programlisting">
sql_query_pre = SET CHARACTER_SET_RESULTS=cp1251
sql_query_pre = SET NAMES utf8
</pre><p>
Also specific to MySQL sources, it is useful to disable query cache
(for indexer connection only) in pre-query, because indexing queries
are not going to be re-run frequently anyway, and there's no sense
in caching their results. That could be achieved with:
</p><pre class="programlisting">
sql_query_pre = SET SESSION query_cache_type=OFF
</pre><p>
</p><h4>Example:</h4><pre class="programlisting">
sql_query_pre = SET NAMES utf8
sql_query_pre = SET SESSION query_cache_type=OFF
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query"></a>11.1.12.&nbsp;sql_query</h3></div></div></div>
<p>
Main document fetch query.
Mandatory, no default value.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
There can be only one main query.
This is the query which is used to retrieve documents from SQL server.
You can specify up to 32 full-text fields (formally, upto SPH_MAX_FIELDS from sphinx.h), and an arbitrary amount of attributes.
All of the columns that are neither document ID (the first one) nor attributes will be full-text indexed.
</p><p>
Document ID <span class="bold"><strong>MUST</strong></span> be the very first field,
and it <span class="bold"><strong>MUST BE UNIQUE UNSIGNED POSITIVE (NON-ZERO, NON-NEGATIVE) INTEGER NUMBER</strong></span>.
It can be either 32-bit or 64-bit, depending on how you built Sphinx;
by default it builds with 32-bit IDs support but <code class="option">--enable-id64</code> option
to <code class="filename">configure</code> allows to build with 64-bit document and word IDs support.

</p><h4>Example:</h4><pre class="programlisting">
sql_query = \
	SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, \
		title, content \
	FROM documents
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-joined-field"></a>11.1.13.&nbsp;sql_joined_field</h3></div></div></div>
<p>
Joined/payload field fetch query.
Multi-value, optional, default is empty list of queries.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
<code class="option">sql_joined_field</code> lets you use two different features:
joined fields, and payloads (payload fields). It's syntax is as follows:
</p><pre class="programlisting">
sql_joined_field = FIELD-NAME 'from'  ( 'query' | 'payload-query' ); QUERY
</pre><p>
where
</p><div class="itemizedlist"><ul type="disc"><li>FIELD-NAME is a joined/payload field name;</li>
<li>QUERY is an SQL query used to fetch values to index.</li>
</ul></div>
<p>
</p><p>
<span class="bold"><strong>Joined fields</strong></span> let you avoid JOIN and/or GROUP_CONCAT statements in the main
document fetch query (sql_query). This can be useful when SQL-side JOIN is slow,
or needs to be offloaded on Sphinx side, or simply to emulate MySQL-specific
GROUP_CONCAT funcionality in case your database server does not support it.
</p><p>
The query must return exactly 2 columns: document ID, and text to append
to a joined field. Document IDs can be duplicate, but they <span class="bold"><strong>must</strong></span> be
in ascending order. All the text rows fetched for a given ID will be
concatented together, and the concatenation result will be indexed
as the entire contents of a joined field. Rows will be concatenated
in the order returned from the query, and separating whitespace
will be inserted between them. For instance, if joined field query
returns the following rows:
</p><pre class="programlisting">
( 1, 'red' )
( 1, 'right' )
( 1, 'hand' )
( 2, 'mysql' )
( 2, 'sphinx' )
</pre><p>
then the indexing results would be equivalent to that of adding
a new text field with a value of 'red right hand' to document 1 and
'mysql sphinx' to document 2.
</p><p>
Joined fields are only indexed differently. There are no other differences
between joined fields and regular text fields.
</p><p>
<span class="bold"><strong>Payloads</strong></span> let you create a special field in which, instead of
keyword positions, so-called user payloads are stored. Payloads are
custom integer values attached to every keyword. They can then be used
in search time to affect the ranking.
</p><p>
The payload query must return exactly 3 columns: document ID; keyword;
and integer payload value. Document IDs can be duplicate, but they <span class="bold"><strong>must</strong></span> be
in ascending order. Payloads must be unsigned integers within 24-bit range,
ie. from 0 to 16777215. For reference, payloads are currently internally
stored as in-field keyword positions, but that is not guaranteed
and might change in the future.
</p><p>
Currently, the only method to account for payloads is to use
SPH_RANK_PROXIMITY_BM25 ranker. On indexes with payload fields,
it will automatically switch to a variant that matches keywords
in those fields, computes a sum of matched payloads multiplied
by field wieghts, and adds that sum to the final rank.
</p><h4>Example:</h4><pre class="programlisting">
sql_joined_field = \
	tagstext from query; \
	SELECT docid, CONCAT('tag',tagid) FROM tags ORDER BY docid ASC
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-range"></a>11.1.14.&nbsp;sql_query_range</h3></div></div></div>
<p>
Range query setup.
Optional, default is empty.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
Setting this option enables ranged document fetch queries (see <a href="#ranged-queries">Section&nbsp;3.7, &#8220;Ranged queries&#8221;</a>).
Ranged queries are useful to avoid notorious MyISAM table locks when indexing
lots of data. (They also help with other less notorious issues, such as reduced
performance caused by big result sets, or additional resources consumed by InnoDB
to serialize big read transactions.)
</p><p>
The query specified in this option must fetch min and max document IDs that will be
used as range boundaries. It must return exactly two integer fields, min ID first
and max ID second; the field names are ignored.
</p><p>
When ranged queries are enabled, <a href="#conf-sql-query" title="11.1.12.&nbsp;sql_query">sql_query</a>
will be required to contain <code class="option">$start</code> and <code class="option">$end</code> macros
(because it obviously would be a mistake to index the whole table many times over).
Note that the intervals specified by <code class="option">$start</code>..<code class="option">$end</code>
will not overlap, so you should <span class="bold"><strong>not</strong></span> remove document IDs that are
exactly equal to <code class="option">$start</code> or <code class="option">$end</code> from your query.
The example in <a href="#ranged-queries">Section&nbsp;3.7, &#8220;Ranged queries&#8221;</a>) illustrates that; note how it
uses greater-or-equal and less-or-equal comparisons.
</p><h4>Example:</h4><pre class="programlisting">
sql_query_range = SELECT MIN(id),MAX(id) FROM documents
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-range-step"></a>11.1.15.&nbsp;sql_range_step</h3></div></div></div>
<p>
Range query step.
Optional, default is 1024.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
Only used when <a href="#ranged-queries">ranged queries</a> are enabled.
The full document IDs interval fetched by <a href="#conf-sql-query-range" title="11.1.14.&nbsp;sql_query_range">sql_query_range</a>
will be walked in this big steps. For example, if min and max IDs fetched
are 12 and 3456 respectively, and the step is 1000, indexer will call
<a href="#conf-sql-query" title="11.1.12.&nbsp;sql_query">sql_query</a> several times with the
following substitutions:
</p><div class="itemizedlist"><ul type="disc"><li>$start=12, $end=1011</li>
<li>$start=1012, $end=2011</li>
<li>$start=2012, $end=3011</li>
<li>$start=3012, $end=3456</li>
</ul></div>
<p>
</p><h4>Example:</h4><pre class="programlisting">
sql_range_step = 1000
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-killlist"></a>11.1.16.&nbsp;sql_query_killlist</h3></div></div></div>
<p>
Kill-list query.
Optional, default is empty (no query).
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 0.9.9-rc1.
</p><p>
This query is expected to return a number of 1-column rows, each containing
just the document ID. The returned document IDs are stored within an index.
Kill-list for a given index suppresses results from <span class="emphasis"><em>other</em></span>
indexes, depending on index order in the query. The intended use is to help
implement deletions and updates on existing indexes without rebuilding
(actually even touching them), and especially to fight phantom results
problem.
</p><p>
Let us dissect an example. Assume we have two indexes, 'main' and 'delta'.
Assume that documents 2, 3, and 5 were deleted since last reindex of 'main',
and documents 7 and 11 were updated (ie. their text contents were changed).
Assume that a keyword 'test' occurred in all these mentioned documents
when we were indexing 'main'; still occurs in document 7 as we index 'delta';
but does not occur in document 11 any more. We now reindex delta and then
search through both these indexes in proper (least to most recent) order:
</p><pre class="programlisting">
$res = $cl-&gt;Query ( "test", "main delta" );
</pre><p>
</p><p>
First, we need to properly handle deletions. The result set should not
contain documents 2, 3, or 5. Second, we also need to avoid phantom results.
Unless we do something about it, document 11 <span class="emphasis"><em>will</em></span>
appear in search results! It will be found in 'main' (but not 'delta').
And it will make it to the final result set unless something stops it.
</p><p>
Kill-list, or K-list for short, is that something. Kill-list attached
to 'delta' will suppress the specified rows from <span class="bold"><strong>all</strong></span> the preceding
indexes, in this case just 'main'. So to get the expected results,
we should put all the updated <span class="emphasis"><em>and</em></span> deleted
document IDs into it.
</p><h4>Example:</h4><pre class="programlisting">
sql_query_killlist = \
	SELECT id FROM documents WHERE updated_ts&gt;=@last_reindex UNION \
	SELECT id FROM documents_deleted WHERE deleted_ts&gt;=@last_reindex
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-uint"></a>11.1.17.&nbsp;sql_attr_uint</h3></div></div></div>
<p>
Unsigned integer <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
The column value should fit into 32-bit unsigned integer range.
Values outside this range will be accepted but wrapped around.
For instance, -1 will be wrapped around to 2^32-1 or 4,294,967,295.
</p><p>
You can specify bit count for integer attributes by appending
':BITCOUNT' to attribute name (see example below).  Attributes with
less than default 32-bit size, or bitfields, perform slower.
But they require less RAM when using <a href="#conf-docinfo" title="11.2.4.&nbsp;docinfo">extern storage</a>:
such bitfields are packed together in 32-bit chunks in <code class="filename">.spa</code>
attribute data file. Bit size settings are ignored if using
<a href="#conf-docinfo" title="11.2.4.&nbsp;docinfo">inline storage</a>.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_uint = group_id
sql_attr_uint = forum_id:9 # 9 bits for forum_id
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-bool"></a>11.1.18.&nbsp;sql_attr_bool</h3></div></div></div>
<p>
Boolean <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Equivalent to <a href="#conf-sql-attr-uint" title="11.1.17.&nbsp;sql_attr_uint">sql_attr_uint</a> declaration with a bit count of 1.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_bool = is_deleted # will be packed to 1 bit
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-bigint"></a>11.1.19.&nbsp;sql_attr_bigint</h3></div></div></div>
<p>
64-bit signed integer <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Note that unlike <a href="#conf-sql-attr-uint" title="11.1.17.&nbsp;sql_attr_uint">sql_attr_uint</a>,
these values are <span class="bold"><strong>signed</strong></span>.
Introduced in version 0.9.9-rc1.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_bigint = my_bigint_id
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-timestamp"></a>11.1.20.&nbsp;sql_attr_timestamp</h3></div></div></div>
<p>
UNIX timestamp <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
The column value should be a timestamp in UNIX format, ie. 32-bit unsigned
integer number of seconds elapsed since midnight, January 01, 1970, GMT.
Timestamps are internally stored and handled as integers everywhere.
But in addition to working with timestamps as integers, it's also legal
to use them along with different date-based functions - such as time segments
sorting mode, or day/week/month/year extraction for GROUP BY. Note that
DATE or DATETIME column types in MySQL can <span class="bold"><strong>not</strong></span> be directly used
as timestamps; you need to explicitly convert such columns using
UNIX_TIMESTAMP function.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_timestamp = UNIX_TIMESTAMP(added_datetime) AS added_ts
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-str2ordinal"></a>11.1.21.&nbsp;sql_attr_str2ordinal</h3></div></div></div>
<p>
Ordinal string number <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
This attribute type (so-called ordinal, for brevity) is intended
to allow sorting by string values, but without storing the strings
themselves. When indexing ordinals, string values are fetched from
database, temporarily stored, sorted, and then replaced by their
respective ordinal numbers in the array of sorted strings.
So, the ordinal number is an integer such that sorting by it
produces the same result as if lexicographically sorting by original strings.
by string values lexicographically.
</p><p>
Earlier versions could consume a lot of RAM for indexing ordinals.
Starting with revision r1112, ordinals accumulation and sorting
also runs in fixed memory (at the cost of using additional temporary
disk space), and honors
<a href="#conf-mem-limit" title="11.3.1.&nbsp;mem_limit">mem_limit</a> settings.
</p><p>
Ideally the strings should be sorted differently, depending
on the encoding and locale. For instance, if the strings are known
to be Russian text in KOI8R encoding, sorting the bytes 0xE0, 0xE1,
and 0xE2 should produce 0xE1, 0xE2 and 0xE0, because in KOI8R
value 0xE0 encodes a character that is (noticeably) after
characters encoded by 0xE1 and 0xE2. Unfortunately, Sphinx
does not support that at the moment and will simply sort
the strings bytewise.
</p><p>
Note that the ordinals are by construction local to each index,
and it's therefore impossible to merge ordinals while retaining
the proper order. The processed strings are replaced by their
sequential number in the index they occurred in, but different
indexes have different sets of strings. For instance, if 'main' index
contains strings "aaa", "bbb", "ccc", and so on up to "zzz",
they'll be assigned numbers 1, 2, 3, and so on up to 26,
respectively. But then if 'delta' only contains "zzz" the assigned
number will be 1. And after the merge, the order will be broken.
Unfortunately, this is impossible to workaround without storing
the original strings (and once Sphinx supports storing the
original strings, ordinals will not be necessary any more). 
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_str2ordinal = author_name
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-float"></a>11.1.22.&nbsp;sql_attr_float</h3></div></div></div>
<p>
Floating point <a href="#attributes" title="3.2.&nbsp;Attributes">attribute</a> declaration.
Multi-value (there might be multiple attributes declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
The values will be stored in single precision, 32-bit IEEE 754 format.
Represented range is approximately from 1e-38 to 1e+38. The amount
of decimal digits that can be stored precisely is approximately 7.
One important usage of the float attributes is storing latitude
and longitude values (in radians), for further usage in query-time
geosphere distance calculations.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_float = lat_radians
sql_attr_float = long_radians
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-multi"></a>11.1.23.&nbsp;sql_attr_multi</h3></div></div></div>
<p>
<a href="#mva" title="3.3.&nbsp;MVA (multi-valued attributes)">Multi-valued attribute</a> (MVA) declaration.
Multi-value (ie. there may be more than one such attribute declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
Plain attributes only allow to attach 1 value per each document.
However, there are cases (such as tags or categories) when it is
desired to attach multiple values of the same attribute and be able
to apply filtering or grouping to value lists.
</p><p>
The declaration format is as follows (backslashes are for clarity only;
everything can be declared in a single line as well):
</p><pre class="programlisting">
sql_attr_multi = ATTR-TYPE ATTR-NAME 'from' SOURCE-TYPE \
	[;QUERY] \
	[;RANGE-QUERY]
</pre><p>
where
</p><div class="itemizedlist"><ul type="disc"><li>ATTR-TYPE is 'uint' or 'timestamp'</li>
<li>SOURCE-TYPE is 'field', 'query', or 'ranged-query'</li>
<li>QUERY is SQL query used to fetch all ( docid, attrvalue ) pairs</li>
<li>RANGE-QUERY is SQL query used to fetch min and max ID values, similar to 'sql_query_range'</li>
</ul></div>
<p>
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_multi = uint tag from query; SELECT id, tag FROM tags
sql_attr_multi = uint tag from ranged-query; \
	SELECT id, tag FROM tags WHERE id&gt;=$start AND id&lt;=$end; \
	SELECT MIN(id), MAX(id) FROM tags
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-string"></a>11.1.24.&nbsp;sql_attr_string</h3></div></div></div>
<p>
String attribute declaration.
Multi-value (ie. there may be more than one such attribute declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 1.10-beta.
</p><p>
String attributes can store arbitrary strings attached to every document.
There's a fixed size limit of 4 MB per value. Also, <code class="filename">searchd</code>
will currently cache all the values in RAM, which is an additional implicit limit.
</p><p>
As of 1.10-beta, strings can only be used for storage and retrieval.
They can not participate in expressions, be used for filtering, sorting,
or grouping (ie. in WHERE, ORDER or GROUP clauses). Note that attributes
declared using <code class="option">sql_attr_string</code> will <span class="bold"><strong>not</strong></span> be full-text
indexed; you can use <a href="#conf-sql-field-string" title="11.1.26.&nbsp;sql_field_string">sql_field_string</a>
directive for that.
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_string = title # will be stored but will not be indexed
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-attr-str2wordcount"></a>11.1.25.&nbsp;sql_attr_str2wordcount</h3></div></div></div>
<p>
Word-count attribute declaration.
Multi-value (ie. there may be more than one such attribute declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 1.10-beta.
</p><p>
Word-count attribute takes a string column, tokenizes it according
to index settings, and stores the resulting number of tokens in an attribute.
This number of tokens ("word count") is a normal integer that can be later
used, for instance, in custom ranking expressions (boost shorter titles,
help identify exact field matches, etc).
</p><h4>Example:</h4><pre class="programlisting">
sql_attr_str2wordcount = title_wc
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-field-string"></a>11.1.26.&nbsp;sql_field_string</h3></div></div></div>
<p>
Combined string attribute and full-text field declaration.
Multi-value (ie. there may be more than one such attribute declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 1.10-beta.
</p><p>
<a href="#conf-sql-attr-string" title="11.1.24.&nbsp;sql_attr_string">sql_attr_string</a> only stores the column
value but does not full-text index it.  In some cases it might be desired to both full-text
index the column and store it as attribute.  <code class="option">sql_field_string</code> lets you do
exactly that. Both the field and the attribute will be named the same.
</p><h4>Example:</h4><pre class="programlisting">
sql_field_string = title # will be both indexed and stored
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-field-str2wordcount"></a>11.1.27.&nbsp;sql_field_str2wordcount</h3></div></div></div>
<p>
Combined word-count attribute and full-text field declaration.
Multi-value (ie. there may be more than one such attribute declared), optional.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 1.10-beta.
</p><a href="#conf-sql-attr-str2wordcount" title="11.1.25.&nbsp;sql_attr_str2wordcount">sql_attr_str2wordcount</a> only stores the column
word count but does not full-text index it.  In some cases it might be desired to both full-text
index the column and also have the count.  <code class="option">sql_field_str2wordcount</code> lets you do
exactly that. Both the field and the attribute will be named the same.
<h4>Example:</h4><pre class="programlisting">
sql_field_str2wordcount = title # will be indexed, and counted/stored
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-file-field"></a>11.1.28.&nbsp;sql_file_field</h3></div></div></div>
<p>
File based field declaration.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 1.10-beta.
</p><p>
This directive makes <code class="filename">indexer</code> interpret field contents
as a file name, and load and index the referred file.  Files larger than
<a href="#conf-max-file-field-buffer" title="11.3.6.&nbsp;max_file_field_buffer">max_file_field_buffer</a>
in size are skipped.  Any errors during the file loading (IO errors, missed
limits, etc) will be reported as indexing warnings and will <span class="bold"><strong>not</strong></span> early
terminate the indexing.  No content will be indexed for such files.
</p><h4>Example:</h4><pre class="programlisting">
sql_file_field = my_file_path # load and index files referred to by my_file_path
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-post"></a>11.1.29.&nbsp;sql_query_post</h3></div></div></div>
<p>
Post-fetch query.
Optional, default value is empty.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
This query is executed immediately after <a href="#conf-sql-query" title="11.1.12.&nbsp;sql_query">sql_query</a>
completes successfully. When post-fetch query produces errors,
they are reported as warnings, but indexing is <span class="bold"><strong>not</strong></span> terminated.
It's result set is ignored. Note that indexing is <span class="bold"><strong>not</strong></span> yet completed
at the point when this query gets executed, and further indexing still may fail.
Therefore, any permanent updates should not be done from here.
For instance, updates on helper table that permanently change
the last successfully indexed ID should not be run from post-fetch
query; they should be run from <a href="#conf-sql-query-post-index" title="11.1.30.&nbsp;sql_query_post_index">post-index query</a> instead.
</p><h4>Example:</h4><pre class="programlisting">
sql_query_post = DROP TABLE my_tmp_table
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-post-index"></a>11.1.30.&nbsp;sql_query_post_index</h3></div></div></div>
<p>
Post-index query.
Optional, default value is empty.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
This query is executed when indexing is fully and succesfully completed.
If this query produces errors, they are reported as warnings,
but indexing is <span class="bold"><strong>not</strong></span> terminated. It's result set is ignored.
<code class="code">$maxid</code> macro can be used in its text; it will be
expanded to maximum document ID which was actually fetched
from the database during indexing.
</p><h4>Example:</h4><pre class="programlisting">
sql_query_post_index = REPLACE INTO counters ( id, val ) \
    VALUES ( 'max_indexed_id', $maxid )
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-ranged-throttle"></a>11.1.31.&nbsp;sql_ranged_throttle</h3></div></div></div>
<p>
Ranged query throttling period, in milliseconds.
Optional, default is 0 (no throttling).
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
</p><p>
Throttling can be useful when indexer imposes too much load on the
database server. It causes the indexer to sleep for given amount of
milliseconds once per each ranged query step. This sleep is unconditional,
and is performed before the fetch query.
</p><h4>Example:</h4><pre class="programlisting">
sql_ranged_throttle = 1000 # sleep for 1 sec before each query step
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-sql-query-info"></a>11.1.32.&nbsp;sql_query_info</h3></div></div></div>
<p>
Document info query.
Optional, default is empty.
Applies to <code class="option">mysql</code> source type only.
</p><p>
Only used by CLI search to fetch and display document information,
only works with MySQL at the moment, and only intended for debugging purposes.
This query fetches the row that will be displayed by CLI search utility
for each document ID. It is required to contain <code class="code">$id</code> macro
that expands to the queried document ID.
</p><h4>Example:</h4><pre class="programlisting">
sql_query_info = SELECT * FROM documents WHERE id=$id
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-command"></a>11.1.33.&nbsp;xmlpipe_command</h3></div></div></div>
<p>
Shell command that invokes xmlpipe stream producer.
Mandatory.
Applies to <code class="option">xmlpipe</code> and <code class="option">xmlpipe2</code> source types only.
</p><p>
Specifies a command that will be executed and which output
will be parsed for documents. Refer to <a href="#xmlpipe" title="3.8.&nbsp;xmlpipe data source">Section&nbsp;3.8, &#8220;xmlpipe data source&#8221;</a>
or <a href="#xmlpipe2" title="3.9.&nbsp;xmlpipe2 data source">Section&nbsp;3.9, &#8220;xmlpipe2 data source&#8221;</a> for specific format description.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_command = cat /home/sphinx/test.xml
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-field"></a>11.1.34.&nbsp;xmlpipe_field</h3></div></div></div>
<p>
xmlpipe field declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only. Refer to <a href="#xmlpipe2" title="3.9.&nbsp;xmlpipe2 data source">Section&nbsp;3.9, &#8220;xmlpipe2 data source&#8221;</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_field = subject
xmlpipe_field = content
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-field-string"></a>11.1.35.&nbsp;xmlpipe_field_string</h3></div></div></div>
<p>
xmlpipe field and string attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only. Refer to <a href="#xmlpipe2" title="3.9.&nbsp;xmlpipe2 data source">Section&nbsp;3.9, &#8220;xmlpipe2 data source&#8221;</a>.
Introduced in version 1.10-beta.
</p><p>
Makes the specified XML element indexed as both a full-text field and a string attribute.
Equivalent to &lt;sphinx:field name="field" attr="string"/&gt; declaration within the XML file.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_field_string = subject
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-field-wordcount"></a>11.1.36.&nbsp;xmlpipe_field_wordcount</h3></div></div></div>
<p>
xmlpipe field and word count attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only. Refer to <a href="#xmlpipe2" title="3.9.&nbsp;xmlpipe2 data source">Section&nbsp;3.9, &#8220;xmlpipe2 data source&#8221;</a>.
Introduced in version 1.10-beta.
</p><p>
Makes the specified XML element indexed as both a full-text field and a word count attribute.
Equivalent to &lt;sphinx:field name="field" attr="wordcount"/&gt; declaration within the XML file.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_field_wordcount = subject
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-uint"></a>11.1.37.&nbsp;xmlpipe_attr_uint</h3></div></div></div>
<p>
xmlpipe integer attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Syntax fully matches that of <a href="#conf-sql-attr-uint" title="11.1.17.&nbsp;sql_attr_uint">sql_attr_uint</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_uint = author
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-bool"></a>11.1.38.&nbsp;xmlpipe_attr_bool</h3></div></div></div>
<p>
xmlpipe boolean attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Syntax fully matches that of <a href="#conf-sql-attr-bool" title="11.1.18.&nbsp;sql_attr_bool">sql_attr_bool</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_bool = is_deleted # will be packed to 1 bit
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-timestamp"></a>11.1.39.&nbsp;xmlpipe_attr_timestamp</h3></div></div></div>
<p>
xmlpipe UNIX timestamp attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Syntax fully matches that of <a href="#conf-sql-attr-timestamp" title="11.1.20.&nbsp;sql_attr_timestamp">sql_attr_timestamp</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_timestamp = published
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-str2ordinal"></a>11.1.40.&nbsp;xmlpipe_attr_str2ordinal</h3></div></div></div>
<p>
xmlpipe string ordinal attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Syntax fully matches that of <a href="#conf-sql-attr-str2ordinal" title="11.1.21.&nbsp;sql_attr_str2ordinal">sql_attr_str2ordinal</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_str2ordinal = author_sort
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-float"></a>11.1.41.&nbsp;xmlpipe_attr_float</h3></div></div></div>
<p>
xmlpipe floating point attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Syntax fully matches that of <a href="#conf-sql-attr-float" title="11.1.22.&nbsp;sql_attr_float">sql_attr_float</a>.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_float = lat_radians
xmlpipe_attr_float = long_radians
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-multi"></a>11.1.42.&nbsp;xmlpipe_attr_multi</h3></div></div></div>
<p>
xmlpipe MVA attribute declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
</p><p>
This setting declares an MVA attribute tag in xmlpipe2 stream.
The contents of the specified tag will be parsed and a list of integers
that will constitute the MVA will be extracted, similar to how
<a href="#conf-sql-attr-multi" title="11.1.23.&nbsp;sql_attr_multi">sql_attr_multi</a> parses
SQL column contents when 'field' MVA source type is specified.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_multi = taglist
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-attr-string"></a>11.1.43.&nbsp;xmlpipe_attr_string</h3></div></div></div>
<p>
xmlpipe string declaration.
Multi-value, optional.
Applies to <code class="option">xmlpipe2</code> source type only.
Introduced in version 1.10-beta.
</p><p>
This setting declares a string attribute tag in xmlpipe2 stream.
The contents of the specified tag will be parsed and stored as a string value.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_attr_string = subject
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-xmlpipe-fixup-utf8"></a>11.1.44.&nbsp;xmlpipe_fixup_utf8</h3></div></div></div>
<p>
Perform Sphinx-side UTF-8 validation and filtering to prevent XML parser from choking on non-UTF-8 documents.
Optional, default is 0.
Applies to <code class="option">xmlpipe2</code> source type only.
</p><p>
Under certain occasions it might be hard or even impossible to guarantee
that the incoming XMLpipe2 document bodies are in perfectly valid and
conforming UTF-8 encoding.  For instance, documents with national
single-byte encodings could sneak into the stream. libexpat XML parser
is fragile, meaning that it will stop processing in such cases.
UTF8 fixup feature lets you avoid that. When fixup is enabled,
Sphinx will preprocess the incoming stream before passing it to the
XML parser and replace invalid UTF-8 sequences with spaces.
</p><h4>Example:</h4><pre class="programlisting">
xmlpipe_fixup_utf8 = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mssql-winauth"></a>11.1.45.&nbsp;mssql_winauth</h3></div></div></div>
<p>
MS SQL Windows authentication flag.
Boolean, optional, default value is 0 (false).
Applies to <code class="option">mssql</code> source type only.
Introduced in version 0.9.9-rc1.
</p><p>
Whether to use currently logged in Windows account credentials for
authentication when connecting to MS SQL Server. Note that when running
<code class="filename">searchd</code> as a service, account user can differ
from the account you used to install the service.
</p><h4>Example:</h4><pre class="programlisting">
mssql_winauth = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mssql-unicode"></a>11.1.46.&nbsp;mssql_unicode</h3></div></div></div>
<p>
MS SQL encoding type flag.
Boolean, optional, default value is 0 (false).
Applies to <code class="option">mssql</code> source type only.
Introduced in version 0.9.9-rc1.
</p><p>
Whether to ask for Unicode or single-byte data when querying MS SQL Server.
This flag <span class="bold"><strong>must</strong></span> be in sync with <a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a> directive;
that is, to index Unicode data, you must set both <code class="option">charset_type</code> in the index
(to 'utf-8') and <code class="option">mssql_unicode</code> in the source (to 1).
For reference, MS SQL will actually return data in UCS-2 encoding instead of UTF-8,
but Sphinx will automatically handle that.
</p><h4>Example:</h4><pre class="programlisting">
mssql_unicode = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-unpack-zlib"></a>11.1.47.&nbsp;unpack_zlib</h3></div></div></div>
<p>
Columns to unpack using zlib (aka deflate, aka gunzip).
Multi-value, optional, default value is empty list of columns.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 0.9.9-rc1.
</p><p>
Columns specified using this directive will be unpacked by <code class="filename">indexer</code>
using standard zlib algorithm (called deflate and also implemented by <code class="filename">gunzip</code>).
When indexing on a different box than the database, this lets you offload the database, and save on network traffic.
The feature is only available if zlib and zlib-devel were both available during build time.
</p><h4>Example:</h4><pre class="programlisting">
unpack_zlib = col1
unpack_zlib = col2
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-unpack-mysqlcompress"></a>11.1.48.&nbsp;unpack_mysqlcompress</h3></div></div></div>
<p>
Columns to unpack using MySQL UNCOMPRESS() algorithm.
Multi-value, optional, default value is empty list of columns.
Applies to SQL source types (<code class="option">mysql</code>, <code class="option">pgsql</code>, <code class="option">mssql</code>) only.
Introduced in version 0.9.9-rc1.
</p><p>
Columns specified using this directive will be unpacked by <code class="filename">indexer</code>
using modified zlib algorithm used by MySQL COMPRESS() and UNCOMPRESS() functions.
When indexing on a different box than the database, this lets you offload the database, and save on network traffic.
The feature is only available if zlib and zlib-devel were both available during build time.
</p><h4>Example:</h4><pre class="programlisting">
unpack_mysqlcompress = body_compressed
unpack_mysqlcompress = description_compressed
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-unpack-mysqlcompress-maxsize"></a>11.1.49.&nbsp;unpack_mysqlcompress_maxsize</h3></div></div></div>
<p>
Buffer size for UNCOMPRESS()ed data.
Optional, default value is 16M.
Introduced in version 0.9.9-rc1.
</p><p>
When using <a href="#conf-unpack-mysqlcompress" title="11.1.48.&nbsp;unpack_mysqlcompress">unpack_mysqlcompress</a>,
due to implementation intrincacies it is not possible to deduce the required buffer size
from the compressed data. So the buffer must be preallocated in advance, and unpacked
data can not go over the buffer size. This option lets you control the buffer size,
both to limit <code class="filename">indexer</code> memory use, and to enable unpacking
of really long data fields if necessary.
</p><h4>Example:</h4><pre class="programlisting">
unpack_mysqlcompress_maxsize = 1M
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="confgroup-index"></a>11.2.&nbsp;Index configuration options</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-index-type"></a>11.2.1.&nbsp;type</h3></div></div></div>
<p>
Index type.
Known values are 'plain', 'distributed', and 'rt'.
Optional, default is 'plain' (plain local index).
</p><p>
Sphinx supports several different types of indexes.
Versions 0.9.x supported two index types: plain local indexes
that are stored and processed on the local machine; and distributed indexes,
that involve not only local searching but querying remote <code class="filename">searchd</code>
instances over the network as well (see <a href="#distributed" title="5.7.&nbsp;Distributed searching">Section&nbsp;5.7, &#8220;Distributed searching&#8221;</a>).
Version 1.10-beta also adds support
for so-called real-time indexes (or RT indexes for short) that
are also stored and processed locally, but additionally allow
for on-the-fly updates of the full-text index (see <a href="#rt-indexes" title="Chapter&nbsp;4.&nbsp;Real-time indexes">Chapter&nbsp;4, <i>Real-time indexes</i></a>).
Note that <span class="emphasis"><em>attributes</em></span> can be updated on-the-fly using
either plain local indexes or RT ones.
</p><p>
Index type setting lets you choose the needed type.
By default, plain local index type will be assumed.
</p><h4>Example:</h4><pre class="programlisting">
type = distributed
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-source"></a>11.2.2.&nbsp;source</h3></div></div></div>
<p>
Adds document source to local index.
Multi-value, mandatory.
</p><p>
Specifies document source to get documents from when the current
index is indexed. There must be at least one source. There may be multiple
sources, without any restrictions on the source types: ie. you can pull
part of the data from MySQL server, part from PostgreSQL, part from
the filesystem using xmlpipe2 wrapper.
</p><p>
However, there are some restrictions on the source data. First,
document IDs must be globally unique across all sources. If that
condition is not met, you might get unexpected search results.
Second, source schemas must be the same in order to be stored
within the same index.
</p><p>
No source ID is stored automatically. Therefore, in order to be able
to tell what source the matched document came from, you will need to
store some additional information yourself. Two typical approaches
include:
</p><div class="orderedlist"><ol type="1"><li>mangling document ID and encoding source ID in it:
<pre class="programlisting">
source src1
{
	sql_query = SELECT id*10+1, ... FROM table1
	...
}

source src2
{
	sql_query = SELECT id*10+2, ... FROM table2
	...
}
</pre></li>
<li>
storing source ID simply as an attribute:
<pre class="programlisting">
source src1
{
	sql_query = SELECT id, 1 AS source_id FROM table1
	sql_attr_uint = source_id
	...
}

source src2
{
	sql_query = SELECT id, 2 AS source_id FROM table2
	sql_attr_uint = source_id
	...
}
</pre></li>
</ol></div>
<p>
</p><h4>Example:</h4><pre class="programlisting">
source = srcpart1
source = srcpart2
source = srcpart3
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-path"></a>11.2.3.&nbsp;path</h3></div></div></div>
<p>
Index files path and file name (without extension).
Mandatory.
</p><p>
Path specifies both directory and file name, but without extension.
<code class="filename">indexer</code> will append different extensions
to this path when generating final names for both permanent and
temporary index files. Permanent data files have several different
extensions starting with '.sp'; temporary files' extensions
start with '.tmp'. It's safe to remove <code class="filename">.tmp*</code>
files is if indexer fails to remove them automatically.
</p><p>
For reference, different index files store the following data:
</p><div class="itemizedlist"><ul type="disc"><li><code class="filename">.spa</code> stores document attributes (used in <a href="#conf-docinfo" title="11.2.4.&nbsp;docinfo">extern docinfo</a> storage mode only);</li>
<li><code class="filename">.spd</code> stores matching document ID lists for each word ID;</li>
<li><code class="filename">.sph</code> stores index header information;</li>
<li><code class="filename">.spi</code> stores word lists (word IDs and pointers to <code class="filename">.spd</code> file);</li>
<li><code class="filename">.spm</code> stores MVA data;</li>
<li><code class="filename">.spp</code> stores hit (aka posting, aka word occurence) lists for each word ID.</li>
</ul></div>
<p>
</p><h4>Example:</h4><pre class="programlisting">
path = /var/data/test1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-docinfo"></a>11.2.4.&nbsp;docinfo</h3></div></div></div>
<p>
Document attribute values (docinfo) storage mode.
Optional, default is 'extern'.
Known values are 'none', 'extern' and 'inline'.
</p><p>
Docinfo storage mode defines how exactly docinfo will be
physically stored on disk and RAM. "none" means that there will be
no docinfo at all (ie. no attributes). Normally you need not to set
"none" explicitly because Sphinx will automatically select "none"
when there are no attributes configured. "inline" means that the
docinfo will be stored in the <code class="filename">.spd</code> file,
along with the document ID lists. "extern" means that the docinfo
will be stored separately (externally) from document ID lists,
in a special <code class="filename">.spa</code> file.
</p><p>
Basically, externally stored docinfo must be kept in RAM when querying.
for performance reasons. So in some cases "inline" might be the only option.
However, such cases are infrequent, and docinfo defaults to "extern".
Refer to <a href="#attributes" title="3.2.&nbsp;Attributes">Section&nbsp;3.2, &#8220;Attributes&#8221;</a> for in-depth discussion
and RAM usage estimates.
</p><h4>Example:</h4><pre class="programlisting">
docinfo = inline
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mlock"></a>11.2.5.&nbsp;mlock</h3></div></div></div>
<p>
Memory locking for cached data.
Optional, default is 0 (do not call mlock()).
</p><p>
For search performance, <code class="filename">searchd</code> preloads
a copy of <code class="filename">.spa</code> and <code class="filename">.spi</code>
files in RAM, and keeps that copy in RAM at all times. But if there
are no searches on the index for some time, there are no accesses
to that cached copy, and OS might decide to swap it out to disk.
First queries to such "cooled down" index will cause swap-in
and their latency will suffer.
</p><p>
Setting mlock option to 1 makes Sphinx lock physical RAM used
for that cached data using mlock(2) system call, and that prevents
swapping (see man 2 mlock for details). mlock(2) is a privileged call,
so it will require <code class="filename">searchd</code> to be either run
from root account, or be granted enough privileges otherwise.
If mlock() fails, a warning is emitted, but index continues
working.
</p><h4>Example:</h4><pre class="programlisting">
mlock = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-morphology"></a>11.2.6.&nbsp;morphology</h3></div></div></div>
<p>
A list of morphology preprocessors to apply.
Optional, default is empty (do not apply any preprocessor).
</p><p>
Morphology preprocessors can be applied to the words being
indexed to replace different forms of the same word with the base,
normalized form. For instance, English stemmer will normalize
both "dogs" and "dog" to "dog", making search results for
both searches the same.
</p><p>
Built-in preprocessors include English stemmer, Russian stemmer
(that supports UTF-8 and Windows-1251 encodings), Soundex,
and Metaphone. The latter two replace the words with special
phonetic codes that are equal is words are phonetically close.
Additional stemmers provided by <a href="http://snowball.tartarus.org/" target="_top">Snowball</a>
project <a href="http://snowball.tartarus.org/dist/libstemmer_c.tgz" target="_top">libstemmer</a> library
can be enabled at compile time using <code class="option">--with-libstemmer</code> <code class="filename">configure</code> option.
Built-in English and Russian stemmers should be faster than their
libstemmer counterparts, but can produce slightly different results,
because they are based on an older version. Metaphone implementation
is based on Double Metaphone algorithm and indexes the primary code.
</p><p>
Built-in values that be used in <code class="option">morphology</code> option are:
'none', 'stem_en', 'stem_ru', 'stem_enru', 'soundex', and 'metaphone'.
Additional values provided by libstemmer are in 'libstemmer_XXX' format,
where XXX is libstemmer algorithm codename (refer to
<code class="filename">libstemmer_c/libstemmer/modules.txt</code> for a complete list).
</p><p>
Several stemmers can be specified (comma-separated). They will be applied
to incoming words in the order they are listed, and the processing will stop
once one of the stemmers actually modifies the word.
Also when <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a> feature is enabled
the word will be looked up in word forms dictionary first, and if there is
a matching entry in the dictionary, stemmers will not be applied at all.
Or in other words, <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a> can be
used to implement stemming exceptions.
</p><h4>Example:</h4><pre class="programlisting">
morphology = stem_en, libstemmer_sv
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-min-stemming-len"></a>11.2.7.&nbsp;min_stemming_len</h3></div></div></div>
<p>
Minimum word length at which to enable stemming.
Optional, default is 1 (stem everything).
Introduced in version 0.9.9-rc1.
</p><p>
Stemmers are not perfect, and might sometimes produce undesired results.
For instance, running "gps" keyword through Porter stemmer for English
results in "gp", which is not really the intent. <code class="option">min_stemming_len</code>
feature lets you suppress stemming based on the source word length,
ie. to avoid stemming too short words. Keywords that are shorter than
the given threshold will not be stemmed. Note that keywords that are
exactly as long as specified <span class="bold"><strong>will</strong></span> be stemmed. So in order to avoid
stemming 3-character keywords, you should specify 4 for the value.
For more finely grained control, refer to <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a> feature.
</p><h4>Example:</h4><pre class="programlisting">
min_stemming_len = 4
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-stopwords"></a>11.2.8.&nbsp;stopwords</h3></div></div></div>
<p>
Stopword files list (space separated).
Optional, default is empty.
</p><p>
Stopwords are the words that will not be indexed. Typically you'd
put most frequent words in the stopwords list because they do not add
much value to search results but consume a lot of resources to process.
</p><p>
You can specify several file names, separated by spaces. All the files
will be loaded. Stopwords file format is simple plain text. The encoding
must match index encoding specified in <a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a>.
File data will be tokenized with respect to <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a>
settings, so you can use the same separators as in the indexed data.
The <a href="#conf-morphology" title="11.2.6.&nbsp;morphology">stemmers</a> will also be
applied when parsing stopwords file.
</p><p>
While stopwords are not indexed, they still do affect the keyword positions.
For instance, assume that "the" is a stopword, that document 1 contains the line
"in office", and that document 2 contains "in the office". Searching for "in office"
as for exact phrase will only return the first document, as expected, even though
"the" in the second one is stopped.
</p><h4>Example:</h4><pre class="programlisting">
stopwords = /usr/local/sphinx/data/stopwords.txt
stopwords = stopwords-ru.txt stopwords-en.txt
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-wordforms"></a>11.2.9.&nbsp;wordforms</h3></div></div></div>
<p>
Word forms dictionary.
Optional, default is empty.
</p><p>
Word forms are applied after tokenizing the incoming text
by <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> rules.
They essentialy let you replace one word with another. Normally,
that would be used to bring different word forms to a single
normal form (eg. to normalize all the variants such as "walks",
"walked", "walking" to the normal form "walk"). It can also be used
to implement stemming exceptions, because stemming is not applied
to words found in the forms list.
</p><p>
Dictionaries are used to normalize incoming words both during indexing
and searching. Therefore, to pick up changes in wordforms file
it's required to reindex and restart <code class="filename">searchd</code>.
</p><p>
Word forms support in Sphinx is designed to support big dictionaries well.
They moderately affect indexing speed: for instance, a dictionary with 1 million
entries slows down indexing about 1.5 times. Searching speed is not affected at all.
Additional RAM impact is roughly equal to the dictionary file size,
and dictionaries are shared across indexes: ie. if the very same 50 MB wordforms
file is specified for 10 different indexes, additional <code class="filename">searchd</code>
RAM usage will be about 50 MB.
</p><p>
Dictionary file should be in a simple plain text format. Each line 
should contain source and destination word forms, in exactly the same
encoding as specified in <a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a>,
separated by "greater" sign. Rules from the
<a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> will be
applied when the file is loaded. So basically it's as case sensitive
as your other full-text indexed data, ie. typically case insensitive.
Here's the file contents sample:
</p><pre class="programlisting">
walks &gt; walk
walked &gt; walk
walking &gt; walk
</pre><p>
</p><p>
There is bundled <code class="filename">spelldump</code> utility that
helps you create a dictionary file in the format Sphinx can read
from source <code class="filename">.dict</code> and <code class="filename">.aff</code>
dictionary files in <code class="filename">ispell</code> or <code class="filename">MySpell</code>
format (as bundled with OpenOffice).
</p><p>
Starting with version 0.9.9-rc1, you can map several source words
to a single destination word. Because the work happens on tokens,
not the source text, differences in whitespace and markup are ignored.
</p><pre class="programlisting">
core 2 duo &gt; c2d
e6600 &gt; c2d
core 2duo &gt; c2d
</pre><p>
</p><h4>Example:</h4><pre class="programlisting">
wordforms = /usr/local/sphinx/data/wordforms.txt
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-exceptions"></a>11.2.10.&nbsp;exceptions</h3></div></div></div>
<p>
Tokenizing exceptions file.
Optional, default is empty.
</p><p>
Exceptions allow to map one or more tokens (including tokens with
characters that would normally be excluded) to a single keyword.
They are similar to <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a>
in that they also perform mapping, but have a number of important
differences.
</p><p>
Short summary of the differences is as follows:
</p><div class="itemizedlist"><ul type="disc"><li>exceptions are case sensitive, wordforms are not;</li>
<li>exceptions allow to detect sequences of tokens, wordforms work with single words only;</li>
<li>exceptions can use special characters that are <span class="bold"><strong>not</strong></span> in charset_table, wordforms fully obey charset_table;</li>
<li>exceptions can underperform on huge dictionaries, wordforms handle millions of entries well.</li>
</ul></div>
<p>
</p><p>
The expected file format is also plain text, with one line per exception,
and the line format is as follows:
</p><pre class="programlisting">
map-from-tokens =&gt; map-to-token
</pre><p>
Example file:
</p><pre class="programlisting">
AT &amp; T =&gt; AT&amp;T
AT&amp;T =&gt; AT&amp;T
Standarten   Fuehrer =&gt; standartenfuhrer
Standarten Fuhrer =&gt; standartenfuhrer
MS Windows =&gt; ms windows
Microsoft Windows =&gt; ms windows
C++ =&gt; cplusplus
c++ =&gt; cplusplus
C plus plus =&gt; cplusplus
</pre><p>
All tokens here are case sensitive: they will <span class="bold"><strong>not</strong></span> be processed by
<a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> rules. Thus, with
the example exceptions file above, "At&amp;t" text will be tokenized as two
keywords "at" and "t", because of lowercase letters. On the other hand,
"AT&amp;T" will match exactly and produce single "AT&amp;T" keyword.
</p><p>
Note that this map-to keyword is a) always interpereted
as a <span class="emphasis"><em>single</em></span> word, and b) is both case and space
sensitive! In our sample, "ms windows" query will <span class="emphasis"><em>not</em></span>
match the document with "MS Windows" text. The query will be interpreted
as a query for two keywords, "ms" and "windows". And what "MS Windows"
gets mapped to is a <span class="emphasis"><em>single</em></span> keyword "ms windows",
with a space in the middle. On the other hand, "standartenfuhrer"
will retrieve documents with "Standarten Fuhrer" or "Standarten Fuehrer"
contents (capitalized exactly like this), or any capitalization variant
of the keyword itself, eg. "staNdarTenfUhreR". (It won't catch
"standarten fuhrer", however: this text does not match any of the
listed exceptions because of case sensitivity, and gets indexed
as two separate keywords.)
</p><p>
Whitespace in the map-from tokens list matters, but its amount does not.
Any amount of the whitespace in the map-form list will match any other amount
of whitespace in the indexed document or query. For instance, "AT&nbsp;&amp;&nbsp;T"
map-from token will match "AT&nbsp;&nbsp;&nbsp;&nbsp;&amp;&nbsp;&nbsp;T" text,
whatever the amount of space in both map-from part and the indexed text.
Such text will therefore be indexed as a special "AT&amp;T" keyword,
thanks to the very first entry from the sample.
</p><p>
Exceptions also allow to capture special characters (that are exceptions
from general <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> rules;
hence the name). Assume that you generally do not want to treat '+'
as a valid character, but still want to be able search for some exceptions
from this rule such as 'C++'. The sample above will do just that, totally
independent of what characters are in the table and what are not.
</p><p>
Exceptions are applied to raw incoming document and query data
during indexing  and searching respectively. Therefore, to pick up
changes in the file it's required to reindex and restart
<code class="filename">searchd</code>.
</p><h4>Example:</h4><pre class="programlisting">
exceptions = /usr/local/sphinx/data/exceptions.txt
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-min-word-len"></a>11.2.11.&nbsp;min_word_len</h3></div></div></div>
<p>
Minimum indexed word length.
Optional, default is 1 (index everything).
</p><p>
Only those words that are not shorter than this minimum will be indexed.
For instance, if min_word_len is 4, then 'the' won't be indexed, but 'they' will be.
</p><h4>Example:</h4><pre class="programlisting">
min_word_len = 4
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-charset-type"></a>11.2.12.&nbsp;charset_type</h3></div></div></div>
<p>
Character set encoding type.
Optional, default is 'sbcs'.
Known values are 'sbcs' and 'utf-8'.
</p><p>
Different encodings have different methods for mapping their internal
characters codes into specific byte sequences. Two most common methods
in use today are single-byte encoding and UTF-8. Their corresponding
charset_type values are 'sbcs' (stands for Single Byte Character Set)
and 'utf-8'. The selected encoding type will be used everywhere where
the index is used: when indexing the data, when parsing the query
against this index, when generating snippets, etc.
</p><p>
Note that while 'utf-8' implies that the decoded values must be treated
as Unicode codepoint numbers, there's a family of 'sbcs' encodings that
may in turn treat different byte values differently, and that should be
properly reflected in your <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a> settings.
For example, the same byte value of 224 (0xE0 hex) maps to different Russian letters
depending on whether koi-8r or windows-1251 encoding is used.
</p><h4>Example:</h4><pre class="programlisting">
charset_type = utf-8
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-charset-table"></a>11.2.13.&nbsp;charset_table</h3></div></div></div>
<p>
Accepted characters table, with case folding rules.
Optional, default value depends on <a href="#conf-charset-type" title="11.2.12.&nbsp;charset_type">charset_type</a> value.
</p><p>
charset_table is the main workhorse of Sphinx tokenizing process,
ie. the process of extracting keywords from document text or query txet.
It controls what characters are accepted as valid and what are not,
and how the accepted characters should be transformed (eg. should
the case be removed or not).
</p><p>
You can think of charset_table as of a big table that has a mapping
for each and every of 100K+ characters in Unicode (or as of a small
256-character table if you're using SBCS). By default, every character
maps to 0, which means that it does not occur within keywords and
should be treated as a separator. Once mentioned in the table,
character is mapped to some other character (most frequently,
either to itself or to a lowercase letter), and is treated
as a valid keyword part.
</p><p>
The expected value format is a commas-separated list of mappings.
Two simplest mappings simply declare a character as valid, and map
a single character to another single character, respectively.
But specifying the whole table in such form would result 
in bloated and barely manageable specifications. So there are
several syntax shortcuts that let you map ranges of characters
at once. The complete list is as follows:
</p><div class="variablelist"><dl><dt><span class="term">A-&gt;a</span></dt>
<dd>Single char mapping, declares source char 'A' as allowed
		to occur within keywords and maps it to destination char 'a'
		(but does <span class="emphasis"><em>not</em></span> declare 'a' as allowed).
	</dd><dt><span class="term">A..Z-&gt;a..z</span></dt>
<dd>Range mapping, declares all chars in source range
		as allowed and maps them to the destination range. Does <span class="emphasis"><em>not</em></span>
		declare destination range as allowed. Also checks ranges' lengths
		(the lengths must be equal).
	</dd><dt><span class="term">a</span></dt>
<dd>Stray char mapping, declares a character as allowed
		and maps it to itself. Equivalent to a-&gt;a single char mapping.
	</dd><dt><span class="term">a..z</span></dt>
<dd>Stray range mapping, declares all characters in range
		as allowed and maps them to themselves. Equivalent to
		a..z-&gt;a..z range mapping.
	</dd><dt><span class="term">A..Z/2</span></dt>
<dd>Checkerboard range map. Maps every pair of chars
		to the second char. More formally, declares odd characters
		in range as allowed and maps them to the even ones; also
		declares even characters as allowed and maps them to themselves.
		For instance, A..Z/2 is equivalent to A-&gt;B, B-&gt;B, C-&gt;D, D-&gt;D,
		..., Y-&gt;Z, Z-&gt;Z. This mapping shortcut is helpful for
		a number of Unicode blocks where uppercase and lowercase
		letters go in such interleaved order instead of contiguous
		chunks.
	</dd></dl></div>
<p>
</p><p>
Control characters with codes from 0 to 31 are always treated as separators.
Characters with codes 32 to 127, ie. 7-bit ASCII characters, can be used
in the mappings as is. To avoid configuration file encoding issues,
8-bit ASCII characters and Unicode characters must be specified in U+xxx form,
where 'xxx' is hexadecimal codepoint number. This form can also be used
for 7-bit ASCII characters to encode special ones: eg. use U+20 to
encode space, U+2E to encode dot, U+2C to encode comma.
</p><h4>Example:</h4><pre class="programlisting">
# 'sbcs' defaults for English and Russian
charset_table = 0..9, A..Z-&gt;a..z, _, a..z, \
	U+A8-&gt;U+B8, U+B8, U+C0..U+DF-&gt;U+E0..U+FF, U+E0..U+FF

# 'utf-8' defaults for English and Russian
charset_table = 0..9, A..Z-&gt;a..z, _, a..z, \
	U+410..U+42F-&gt;U+430..U+44F, U+430..U+44F
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-ignore-chars"></a>11.2.14.&nbsp;ignore_chars</h3></div></div></div>
<p>
Ignored characters list.
Optional, default is empty.
</p><p>
Useful in the cases when some characters, such as soft hyphenation mark (U+00AD),
should be not just treated as separators but rather fully ignored.
For example, if '-' is simply not in the charset_table,
"abc-def" text will be indexed as "abc" and "def" keywords.
On the contrary, if '-' is added to ignore_chars list, the same
text will be indexed as a single "abcdef" keyword.
</p><p>
The syntax is the same as for <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a>,
but it's only allowed to declare characters, and not allowed to map them. Also,
the ignored characters must not be present in charset_table.
</p><h4>Example:</h4><pre class="programlisting">
ignore_chars = U+AD
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-min-prefix-len"></a>11.2.15.&nbsp;min_prefix_len</h3></div></div></div>
<p>
Minimum word prefix length to index.
Optional, default is 0 (do not index prefixes).
</p><p>
Prefix indexing allows to implement wildcard searching by 'wordstart*' wildcards
(refer to <a href="#conf-enable-star" title="11.2.19.&nbsp;enable_star">enable_star</a> option for details on wildcard syntax).
When mininum prefix length is set to a positive number, indexer will index
all the possible keyword prefixes (ie. word beginnings) in addition to the keywords
themselves. Too short prefixes (below the minimum allowed length) will not
be indexed.
</p><p>
For instance, indexing a keyword "example" with min_prefix_len=3
will result in indexing "exa", "exam", "examp", "exampl" prefixes along
with the word itself. Searches against such index for "exam" will match
documents that contain "example" word, even if they do not contain "exam"
on itself. However, indexing prefixes will make the index grow significantly
(because of many more indexed keywords), and will degrade both indexing
and searching times.
</p><p>
There's no automatic way to rank perfect word matches higher
in a prefix index, but there's a number of tricks to achieve that.
First, you can setup two indexes, one with prefix indexing and one
without it, search through both, and use <a href="#api-func-setindexweights" title="8.3.6.&nbsp;SetIndexWeights">SetIndexWeights()</a>
call to combine weights. Second, you can enable star-syntax and rewrite
your extended-mode queries:
</p><pre class="programlisting">
# in sphinx.conf
enable_star = 1

// in query
$cl-&gt;Query ( "( keyword | keyword* ) other keywords" );
</pre><p>
</p><h4>Example:</h4><pre class="programlisting">
min_prefix_len = 3
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-min-infix-len"></a>11.2.16.&nbsp;min_infix_len</h3></div></div></div>
<p>
Minimum infix prefix length to index.
Optional, default is 0 (do not index infixes).
</p>
Infix indexing allows to implement wildcard searching by 'start*', '*end', and '*middle*' wildcards
(refer to <a href="#conf-enable-star" title="11.2.19.&nbsp;enable_star">enable_star</a> option for details on wildcard syntax).
When mininum infix length is set to a positive number, indexer will index all the possible keyword infixes
(ie. substrings) in addition to the keywords themselves. Too short infixes
(below the minimum allowed length) will not be indexed.
<p>
For instance, indexing a keyword "test" with min_infix_len=2
will result in indexing "te", "es", "st", "tes", "est" infixes along
with the word itself. Searches against such index for "es" will match
documents that contain "test" word, even if they do not contain "es"
on itself. However, indexing infixes will make the index grow significantly
(because of many more indexed keywords), and will degrade both indexing
and searching times.
</p><p>
There's no automatic way to rank perfect word matches higher
in an infix index, but the same tricks as with <a href="#conf-min-prefix-len" title="11.2.15.&nbsp;min_prefix_len">prefix indexes</a>
can be applied.
</p><h4>Example:</h4><pre class="programlisting">
min_infix_len = 3
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-prefix-fields"></a>11.2.17.&nbsp;prefix_fields</h3></div></div></div>
<p>
The list of full-text fields to limit prefix indexing to.
Optional, default is empty (index all fields in prefix mode).
</p><p>
Because prefix indexing impacts both indexing and searching performance,
it might be desired to limit it to specific full-text fields only:
for instance, to provide prefix searching through URLs, but not through
page contents. prefix_fields specifies what fields will be prefix-indexed;
all other fields will be indexed in normal mode. The value format is a
comma-separated list of field names.
</p><h4>Example:</h4><pre class="programlisting">
prefix_fields = url, domain
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-infix-fields"></a>11.2.18.&nbsp;infix_fields</h3></div></div></div>
<p>
The list of full-text fields to limit infix indexing to.
Optional, default is empty (index all fields in infix mode).
</p><p>
Similar to <a href="#conf-prefix-fields" title="11.2.17.&nbsp;prefix_fields">prefix_fields</a>,
but lets you limit infix-indexing to given fields.
</p><h4>Example:</h4><pre class="programlisting">
infix_fields = url, domain
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-enable-star"></a>11.2.19.&nbsp;enable_star</h3></div></div></div>
<p>
Enables star-syntax (or wildcard syntax) when searching through prefix/infix indexes.
Optional, default is is 0 (do not use wildcard syntax), for compatibility with 0.9.7.
Known values are 0 and 1.
</p><p>
This feature enables "star-syntax", or wildcard syntax, when searching
through indexes which were created with prefix or infix indexing enabled.
It only affects searching; so it can be changed without reindexing
by simply restarting <code class="filename">searchd</code>.
</p><p>
The default value is 0, that means to disable star-syntax
and treat all keywords as prefixes or infixes respectively,
depending on indexing-time <a href="#conf-min-prefix-len" title="11.2.15.&nbsp;min_prefix_len">min_prefix_len</a>
and <a href="#conf-min-infix-len" title="11.2.16.&nbsp;min_infix_len">min_infix_len settings</a>.
The value of 1 means that star ('*') can be used at the start
and/or the end of the keyword. The star will match zero or more characters.
</p><p>
For example, assume that the index was built with infixes and
that enable_star is 1. Searching should work as follows:
</p><div class="orderedlist"><ol type="1"><li>"abcdef" query will match only those documents that contain the exact "abcdef" word in them.</li>
<li>"abc*" query will match those documents that contain
any words starting with "abc" (including the documents which
contain the exact "abc" word only);</li>
<li>"*cde*" query will match those documents that contain
any words which have "cde" characters in any part of the word
(including the documents which contain the exact "cde" word only).</li>
<li>"*def" query will match those documents that contain
any words ending with "def" (including the documents that
contain the exact "def" word only).</li>
</ol></div>
<p>
</p><h4>Example:</h4><pre class="programlisting">
enable_star = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-ngram-len"></a>11.2.20.&nbsp;ngram_len</h3></div></div></div>
<p>
N-gram lengths for N-gram indexing.
Optional, default is 0 (disable n-gram indexing).
Known values are 0 and 1 (other lengths to be implemented).
</p><p>
N-grams provide basic CJK (Chinese, Japanse, Koreasn) support for
unsegmented texts. The issue with CJK searching is that there could be no
clear separators between the words. Ideally, the texts would be filtered
through a special program called segmenter that would insert separators
in proper locations. However, segmenters are slow and error prone,
and it's common to index contiguous groups of N characters, or n-grams,
instead.
</p><p>
When this feature is enabled, streams of CJK characters are indexed
as N-grams. For example, if incoming text is "ABCDEF" (where A to F represent
some CJK characters) and length is 1, in will be indexed as if
it was "A B C D E F". (With length equal to 2, it would produce "AB BC CD DE EF";
but only 1 is supported at the moment.) Only those characters that are
listed in <a href="#conf-ngram-chars" title="11.2.21.&nbsp;ngram_chars">ngram_chars</a> table
will be split this way; other ones will not be affected.
</p><p>
Note that if search query is segmented, ie. there are separators between
individual words, then wrapping the words in quotes and using extended mode
will resut in proper matches being found even if the text was <span class="bold"><strong>not</strong></span>
segmented. For instance, assume that the original query is BC&nbsp;DEF.
After wrapping in quotes on the application side, it should look
like "BC"&nbsp;"DEF" (<span class="emphasis"><em>with</em></span> quotes). This query
will be passed to Sphinx and internally split into 1-grams too,
resulting in "B&nbsp;C"&nbsp;"D&nbsp;E&nbsp;F" query, still with
quotes that are the phrase matching operator. And it will match
the text even though there were no separators in the text.
</p><p>
Even if the search query is not segmented, Sphinx should still produce
good results, thanks to phrase based ranking: it will pull closer phrase
matches (which in case of N-gram CJK words can mean closer multi-character
word matches) to the top.
</p><h4>Example:</h4><pre class="programlisting">
ngram_len = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-ngram-chars"></a>11.2.21.&nbsp;ngram_chars</h3></div></div></div>
<p>
N-gram characters list.
Optional, default is empty.
</p><p>
To be used in conjunction with in <a href="#conf-ngram-len" title="11.2.20.&nbsp;ngram_len">ngram_len</a>,
this list defines characters, sequences of which are subject to N-gram extraction.
Words comprised of other characters will not be affected by N-gram indexing
feature. The value format is identical to <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a>.
</p><h4>Example:</h4><pre class="programlisting">
ngram_chars = U+3000..U+2FA1F
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-phrase-boundary"></a>11.2.22.&nbsp;phrase_boundary</h3></div></div></div>
<p>
Phrase boundary characters list.
Optional, default is empty.
</p><p>
This list controls what characters will be treated as phrase boundaries,
in order to adjust word positions and enable phrase-level search
emulation through proximity search. The syntax is similar
to <a href="#conf-charset-table" title="11.2.13.&nbsp;charset_table">charset_table</a>.
Mappings are not allowed and the boundary characters must not
overlap with anything else.
</p><p>
On phrase boundary, additional word position increment (specified by
<a href="#conf-phrase-boundary-step" title="11.2.23.&nbsp;phrase_boundary_step">phrase_boundary_step</a>)
will be added to current word position. This enables phrase-level
searching through proximity queries: words in different phrases
will be guaranteed to be more than phrase_boundary_step distance
away from each other; so proximity search within that distance
will be equivalent to phrase-level search.
</p><p>
Phrase boundary condition will be raised if and only if such character
is followed by a separator; this is to avoid abbreviations such as
S.T.A.L.K.E.R or URLs being treated as several phrases.
</p><h4>Example:</h4><pre class="programlisting">
phrase_boundary = ., ?, !, U+2026 # horizontal ellipsis
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-phrase-boundary-step"></a>11.2.23.&nbsp;phrase_boundary_step</h3></div></div></div>
<p>
Phrase boundary word position increment.
Optional, default is 0.
</p><p>
On phrase boundary, current word position will be additionally incremented
by this number. See <a href="#conf-phrase-boundary" title="11.2.22.&nbsp;phrase_boundary">phrase_boundary</a> for details.
</p><h4>Example:</h4><pre class="programlisting">
phrase_boundary_step = 100
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-html-strip"></a>11.2.24.&nbsp;html_strip</h3></div></div></div>
<p>
Whether to strip HTML markup from incoming full-text data.
Optional, default is 0.
Known values are 0 (disable stripping) and 1 (enable stripping).
</p><p>
Stripping does not work with <code class="option">xmlpipe</code> source type
(it's suggested to upgrade to xmlpipe2 anyway). It should work with
properly formed HTML and XHTML, but, just as most browsers, may produce
unexpected results on malformed input (such as HTML with stray &lt;'s
or unclosed &gt;'s).
</p><p>
Only the tags themselves, and also HTML comments, are stripped.
To strip the contents of the tags too (eg. to strip embedded scripts),
see <a href="#conf-html-remove-elements" title="11.2.26.&nbsp;html_remove_elements">html_remove_elements</a> option.
There are no restrictions on tag names; ie. everything
that looks like a valid tag start, or end, or a comment
will be stripped.
</p><h4>Example:</h4><pre class="programlisting">
html_strip = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-html-index-attrs"></a>11.2.25.&nbsp;html_index_attrs</h3></div></div></div>
<p>
A list of markup attributes to index when stripping HTML.
Optional, default is empty (do not index markup attributes).
</p><p>
Specifies HTML markup attributes whose contents should be retained and indexed
even though other HTML markup is stripped. The format is per-tag enumeration of
indexable attributes, as shown in the example below.
</p><h4>Example:</h4><pre class="programlisting">
html_index_attrs = img=alt,title; a=title;
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-html-remove-elements"></a>11.2.26.&nbsp;html_remove_elements</h3></div></div></div>
<p>
A list of HTML elements for which to strip contents along with the elements themselves.
Optional, default is empty string (do not strip contents of any elements).
</p><p>
This feature allows to strip element contents, ie. everything that
is between the opening and the closing tags. It is useful to remove
embedded scripts, CSS, etc.	Short tag form for empty elements
(ie. &lt;br /&gt;) is properly supported; ie. the text that
follows such tag will <span class="bold"><strong>not</strong></span> be removed.
</p><p>
The value is a comma-separated list of element (tag) names whose
contents should be removed. Tag names are case insensitive.
</p><h4>Example:</h4><pre class="programlisting">
html_remove_elements = style, script
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-local"></a>11.2.27.&nbsp;local</h3></div></div></div>
<p>
Local index declaration in the <a href="#distributed" title="5.7.&nbsp;Distributed searching">distributed index</a>.
Multi-value, optional, default is empty.
</p><p>
This setting is used to declare local indexes that will be searched when
given distributed index is searched. All local indexes will be searched
<span class="bold"><strong>sequentially</strong></span>, utilizing only 1 CPU or core; to parallelize processing,
you can configure <code class="filename">searchd</code> to query itself (refer to
<a href="#conf-agent" title="11.2.28.&nbsp;agent">Section&nbsp;11.2.28, &#8220;agent&#8221;</a> for the details). There might be several local
indexes declared per each distributed index. Any local index can be mentioned
several times in other distributed indexes.
</p><h4>Example:</h4><pre class="programlisting">
local = chunk1
local = chunk2
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-agent"></a>11.2.28.&nbsp;agent</h3></div></div></div>
<p>
Remote agent declaration in the <a href="#distributed" title="5.7.&nbsp;Distributed searching">distributed index</a>.
Multi-value, optional, default is empty.
</p><p>
This setting is used to declare remote agents that will be searched
when given distributed index is searched. The agents can be thought of
as network pointers that specify host, port, and index names. In the basic
case agents would correspond to remote physical machines. More formally,
that is not always correct: you can point several agents to the
same remote machine; or you can even point agents to the very same
single instance of <code class="filename">searchd</code> (in order to utilize
many CPUs or cores).
</p><p>
The value format is as follows:
</p><pre class="programlisting">
agent = specification:remote-indexes-list
specification = hostname ":" port | path
</pre><p>
Where 'hostname' is remote host name; 'port' is remote TCP port; 'path'
is Unix-domain socket path and 'remote-indexes-list' is a
comma-separated list of remote index names.
</p><p>
All agents will be searched in parallel. However, all indexes
specified for a given agent will be searched sequentially
in this agent. This lets you fine-tune the configuration
to the hardware. For instance, if two remote indexes are stored
on the same physical HDD, it's better to configure one agent
with several sequentially searched indexes to avoid HDD steping.
If they are stored on different HDDs, having two agents will be
advantageous, because the work will be fully parallelized.
The same applies to CPUs; though CPU performance impact caused
by two processes stepping on each other is somewhat smaller
and frequently can be ignored at all.
</p><p>
On machines with many CPUs and/or HDDs, agents can be pointed
to the same machine to utilize all of the hardware in parallel
and reduce query latency. There is no need to setup several
<code class="filename">searchd</code> instances for that; it's legal
to configure the instance to contact itself. Here's an example
setup, intended for a 4-CPU machine, that will use up to
4 CPUs in parallel to process each query:
</p><pre class="programlisting">
index dist
{
	type = distributed
	local = chunk1
	agent = localhost:9312:chunk2
	agent = localhost:9312:chunk3
	agent = localhost:9312:chunk4
}
</pre><p>
Note how one of the chunks is searched locally and the same instance
of searchd queries itself to launch searches through three other ones
in parallel.
</p><h4>Example:</h4><pre class="programlisting">
agent = localhost:9312:chunk2 # contact itself
agent = /var/run/searchd.s:chunk2
agent = searchbox2:9312:chunk3,chunk4 # search remote indexes
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-agent-blackhole"></a>11.2.29.&nbsp;agent_blackhole</h3></div></div></div>
<p>
Remote blackhole agent declaration in the <a href="#distributed" title="5.7.&nbsp;Distributed searching">distributed index</a>.
Multi-value, optional, default is empty.
Introduced in version 0.9.9-rc1.
</p><p>
<code class="option">agent_blackhole</code> lets you fire-and-forget queries
to remote agents. That is useful for debugging (or just testing)
production clusters: you can setup a separate debugging/testing searchd
instance, and forward the requests to this instance from your production
master (aggregator) instance without interfering with production work.
Master searchd will attempt to connect and query blackhole agent
normally, but it will neither wait nor process any responses.
Also, all network errors on blackhole agents will be ignored.
The value format is completely identical to regular
<a href="#conf-agent" title="11.2.28.&nbsp;agent">agent</a> directive.
</p><h4>Example:</h4><pre class="programlisting">
agent_blackhole = testbox:9312:testindex1,testindex2
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-agent-connect-timeout"></a>11.2.30.&nbsp;agent_connect_timeout</h3></div></div></div>
<p>
Remote agent connection timeout, in milliseconds.
Optional, default is 1000 (ie. 1 second).
</p><p>
When connecting to remote agents, <code class="filename">searchd</code>
will wait at most this much time for connect() call to complete
succesfully. If the timeout is reached but connect() does not complete,
and <a href="#api-func-setretries" title="8.1.4.&nbsp;SetRetries">retries</a> are enabled,
retry will be initiated.
</p><h4>Example:</h4><pre class="programlisting">
agent_connect_timeout = 300
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-agent-query-timeout"></a>11.2.31.&nbsp;agent_query_timeout</h3></div></div></div>
<p>
Remote agent query timeout, in milliseconds.
Optional, default is 3000 (ie. 3 seconds).
</p><p>
After connection, <code class="filename">searchd</code> will wait at most this
much time for remote queries to complete. This timeout is fully separate
from connection timeout; so the maximum possible delay caused by
a remote agent equals to the sum of <code class="code">agent_connection_timeout</code> and
<code class="code">agent_query_timeout</code>. Queries will <span class="bold"><strong>not</strong></span> be retried
if this timeout is reached; a warning will be produced instead.
</p><h4>Example:</h4><pre class="programlisting">
agent_query_timeout = 10000 # our query can be long, allow up to 10 sec
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-preopen"></a>11.2.32.&nbsp;preopen</h3></div></div></div>
<p>
Whether to pre-open all index files, or open them per each query.
Optional, default is 0 (do not preopen).
</p><p>
This option tells <code class="filename">searchd</code> that it should pre-open
all index files on startup (or rotation) and keep them open while it runs.
Currently, the default mode is <span class="bold"><strong>not</strong></span> to pre-open the files (this may
change in the future). Preopened indexes take a few (currently 2) file
descriptors per index. However, they save on per-query <code class="code">open()</code> calls;
and also they are invulnerable to subtle race conditions that may happen during
index rotation under high load. On the other hand, when serving many indexes
(100s to 1000s), it still might be desired to open the on per-query basis
in order to save file descriptors.
</p><p>
This directive does not affect <code class="filename">indexer</code> in any way,
it only affects <code class="filename">searchd</code>.
</p><h4>Example:</h4><pre class="programlisting">
preopen = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-ondisk-dict"></a>11.2.33.&nbsp;ondisk_dict</h3></div></div></div>
<p>
Whether to keep the dictionary file (.spi) for this index on disk, or precache it in RAM.
Optional, default is 0 (precache in RAM).
Introduced in version 0.9.9-rc1.
</p><p>
The dictionary (.spi) can be either kept on RAM or on disk. The default
is to fully cache it in RAM. That improves performance, but might cause
too much RAM pressure, especially if prefixes or infixes were used.
Enabling <code class="option">ondisk_dict</code> results in 1 additional disk IO
per keyword per query, but reduces memory footprint.
</p><p>
This directive does not affect <code class="filename">indexer</code> in any way,
it only affects <code class="filename">searchd</code>.
</p><h4>Example:</h4><pre class="programlisting">
ondisk_dict = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-inplace-enable"></a>11.2.34.&nbsp;inplace_enable</h3></div></div></div>
<p>
Whether to enable in-place index inversion.
Optional, default is 0 (use separate temporary files).
Introduced in version 0.9.9-rc1.
</p><p>
<code class="option">inplace_enable</code> greatly reduces indexing disk footprint,
at a cost of slightly slower indexing (it uses around 2x less disk,
but yields around 90-95% the original performance).
</p><p>
Indexing involves two major phases. The first phase collects,
processes, and partially sorts documents by keyword, and writes
the intermediate result to temporary files (.tmp*). The second
phase fully sorts the documents, and creates the final index
files. Thus, rebuilding a production index on the fly involves
around 3x peak disk footprint: 1st copy for the intermediate
temporary files, 2nd copy for newly constructed copy, and 3rd copy
for the old index that will be serving production queries in the meantime.
(Intermediate data is comparable in size to the final index.)
That might be too much disk footprint for big data collections,
and <code class="option">inplace_enable</code> allows to reduce it.
When enabled, it reuses the temporary files, outputs the
final data back to them, and renames them on completion.
However, this might require additional temporary data chunk
relocation, which is where the performance impact comes from.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
inplace_enable = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-inplace-hit-gap"></a>11.2.35.&nbsp;inplace_hit_gap</h3></div></div></div>
<p>
<a href="#conf-inplace-enable" title="11.2.34.&nbsp;inplace_enable">In-place inversion</a> fine-tuning option.
Controls preallocated hitlist gap size.
Optional, default is 0.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
inplace_hit_gap = 1M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-inplace-docinfo-gap"></a>11.2.36.&nbsp;inplace_docinfo_gap</h3></div></div></div>
<p>
<a href="#conf-inplace-enable" title="11.2.34.&nbsp;inplace_enable">In-place inversion</a> fine-tuning option.
Controls preallocated docinfo gap size.
Optional, default is 0.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
inplace_docinfo_gap = 1M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-inplace-reloc-factor"></a>11.2.37.&nbsp;inplace_reloc_factor</h3></div></div></div>
<p>
<a href="#conf-inplace-reloc-factor" title="11.2.37.&nbsp;inplace_reloc_factor">In-place inversion</a> fine-tuning option.
Controls relocation buffer size within indexing memory arena.
Optional, default is 0.1.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
inplace_reloc_factor = 0.1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-inplace-write-factor"></a>11.2.38.&nbsp;inplace_write_factor</h3></div></div></div>
<p>
<a href="#conf-inplace-write-factor" title="11.2.38.&nbsp;inplace_write_factor">In-place inversion</a> fine-tuning option.
Controls in-place write buffer size within indexing memory arena.
Optional, default is 0.1.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
inplace_write_factor = 0.1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-index-exact-words"></a>11.2.39.&nbsp;index_exact_words</h3></div></div></div>
<p>
Whether to index the original keywords along with the stemmed/remapped versions.
Optional, default is 0 (do not index).
Introduced in version 0.9.9-rc1.
</p><p>
When enabled, <code class="option">index_exact_words</code> forces <code class="filename">indexer</code>
to put the raw keywords in the index along with the stemmed versions. That, in turn,
enables <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">exact form operator</a> in the query language to work.
This impacts the index size and the indexing time. However, searching performance
is not impacted at all.
</p><h4>Example:</h4><pre class="programlisting">
index_exact_words = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-overshort-step"></a>11.2.40.&nbsp;overshort_step</h3></div></div></div>
<p>
Position increment on overshort (less that <a href="#conf-min-word-len" title="11.2.11.&nbsp;min_word_len">min_word_len</a>) keywords.
Optional, allowed values are 0 and 1, default is 1.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
overshort_step = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-stopword-step"></a>11.2.41.&nbsp;stopword_step</h3></div></div></div>
<p>
Position increment on <a href="#conf-stopwords" title="11.2.8.&nbsp;stopwords">stopwords</a>.
Optional, allowed values are 0 and 1, default is 1.
Introduced in version 0.9.9-rc1.
</p><p>
This directive does not affect <code class="filename">searchd</code> in any way,
it only affects <code class="filename">indexer</code>.
</p><h4>Example:</h4><pre class="programlisting">
stopword_step = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-hitless-words"></a>11.2.42.&nbsp;hitless_words</h3></div></div></div>
<p>
Hitless words list.
Optional, allowed values are 'all', or a list file name.
Introduced in version 1.10-beta.
</p><p>
By default, Sphinx full-text index stores not only a list of matching
documents for every given keyword, but also a list of its in-document positions
(aka hitlist). Hitlists enables phrase, proximity, strict order and other
advanced types of searching, as well as phrase proximity ranking. However,
hitlists for specific frequent keywords (that can not be stopped for
some reason despite being frequent) can get huge and thus slow to process
while querying. Also, in some cases we might only care about boolean
keyword matching, and never need position-based searching operators
(such as phrase matching) nor phrase ranking.
</p><p>
<code class="option">hitless_words</code> lets you create indexes that either
do not have positional information (hitlists) at all, or skip it for
specific keywords.
</p><p>
Hitless index will generally use less space than the respective
regular index (about 1.5x can be expected). Both indexing and searching
should be faster, at a cost of missing positional query and ranking support.
When searching, positional queries (eg. phrase queries) will be automatically
converted to respective non-positional (document-level) or combined queries.
For instance, if keywords "hello" and "world" are hitless, "hello world"
phrase query will be converted to (hello &amp; world) bag-of-words query,
matching all documents that mention either of the keywords but not necessarily
the exact phrase. And if, in addition, keywords "simon" and "says" are not
hitless, "simon says hello world" will be converted to ("simon says" &amp;
hello &amp; world) query, matching all documents that contain "hello" and
"world" anywhere in the document, and also "simon says" as an exact phrase.
</p><h4>Example:</h4><pre class="programlisting">
hitless_words = all
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-expand-keywords"></a>11.2.43.&nbsp;expand_keywords</h3></div></div></div>
<p>
Expand keywords with exact forms and/or stars when possible.
Optional, default is 0 (do not expand keywords).
Introduced in version 1.10-beta.
</p><p>
Queries against indexes with <code class="option">expand_keywords</code> feature
enabled are internally expanded as follows. If the index was built with
prefix or infix indexing enabled, every keyword gets internally replaced
with a disjunction of keyword itself and a respective prefix or infix
(keyword with stars). If the index was built with both stemming and
<a href="#conf-index-exact-words" title="11.2.39.&nbsp;index_exact_words">index_exact_words</a> enabled,
exact form is also added. Here's an example that shows how internal
expansion works when all of the above (infixes, stemming, and exact
words) are combined:
</p><pre class="programlisting">
running -&gt; ( running | *running* | =running )
</pre><p>
</p><p>
Expanded queries take naturally longer to complete, but can possibly
improve the search quality, as the documents with exact form matches
should be ranked generally higher than documents with stemmed or infix matches.
</p><p>
Note that the existing query syntax does not allowe to emulate this
kind of expansion, because internal expansion works on keyword level and
expands keywords within phrase or quorum operators too (which is not
possible through the query syntax).
</p><p>
This directive does not affect <code class="filename">indexer</code> in any way,
it only affects <code class="filename">searchd</code>.
</p><h4>Example:</h4><pre class="programlisting">
expand_keywords = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-blend-chars"></a>11.2.44.&nbsp;blend_chars</h3></div></div></div>
<p>
Blended characters list.
Optional, default is empty.
Introduced in version 1.10-beta.
</p><p>
Blended characters are indexed both as separators and valid characters.
For instance, assume that &amp; is configured as blended and AT&amp;T
occurs in an indexed document. Three different keywords will get indexed,
namely "at&amp;t", treating blended characters as valid, plus "at" and "t",
treating them as separators. Note that all these three keywords will be
assigned the very same position (in a sense blending with each other).
Thus, querying for either AT&amp;T or just AT will match that document.
However, querying for "AT T" phrase will not match it.
</p><p>
Blended characters can overlap with special characters used in query
syntax (think of T-Mobile or @twitter). Where possible, query parser will
automatically handle blended character as blended. For instance, "hello @twitter"
within quotes (a phrase operator) would handle @-sign as blended, because
@-syntax for field operator is not allowed within phrases. Otherwise,
the character would be handled as an operator. So you might want to
escape the query terms.
</p><h4>Example:</h4><pre class="programlisting">
blend_chars = +, &amp;, U+23
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-mem-limit"></a>11.2.45.&nbsp;rt_mem_limit</h3></div></div></div>
<p>
RAM chunk size limit.
Optional, default is empty.
Introduced in version 1.10-beta.
</p><p>
RT index keeps some data in memory (so-called RAM chunk) and
also maintains a number of on-disk indexes (so-called disk chunks).
This directive lets you control the RAM chunk size. Once there's
too much data to keep in RAM, RT index will flush it to disk,
activate a newly created disk chunk, and reset the RAM chunk.
</p><p>
The limit is pretty strict; RT index should never allocate more
memory than it's limited to. The memory is not preallocated either,
hence, specifying 512 MB limit and only inserting 3 MB of data
should result in allocating 3 MB, not 512 MB.
</p><p>
</p><h4>Example:</h4><pre class="programlisting">
rt_mem_limit = 512M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-field"></a>11.2.46.&nbsp;rt_field</h3></div></div></div>
<p>
Full-text field declaration.
Multi-value, mandatory
Introduced in version 1.10-beta.
</p><p>
Full-text fields to be indexed are declared using <code class="option">rt_field</code>
directive. The names must be unique. The order is preserved; and so field values
in INSERT statements without an explicit list of inserted columns will have to be
in the same order as configured.
</p><p>
</p><h4>Example:</h4><pre class="programlisting">
rt_field = author
rt_field = title
rt_field = content
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-attr-uint"></a>11.2.47.&nbsp;rt_attr_uint</h3></div></div></div>
<p>
Unsigned integer attribute declaration.
Multi-value (an arbitrary number of attributes is allowed), optional.
Declares an unsigned 32-bit attribute.
Introduced in version 1.10-beta.
</p><h4>Example:</h4><pre class="programlisting">
rt_attr_uint = gid
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-attr-bigint"></a>11.2.48.&nbsp;rt_attr_bigint</h3></div></div></div>
<p>
BIGINT attribute declaration.
Multi-value (an arbitrary number of attributes is allowed), optional.
Declares a signed 64-bit attribute.
Introduced in version 1.10-beta.
</p><h4>Example:</h4><pre class="programlisting">
rt_attr_bigint = guid
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-attr-float"></a>11.2.49.&nbsp;rt_attr_float</h3></div></div></div>
<p>
Floating point attribute declaration.
Multi-value (an arbitrary number of attributes is allowed), optional.
Declares a single precision, 32-bit IEEE 754 format float attribute.
Introduced in version 1.10-beta.
</p><h4>Example:</h4><pre class="programlisting">
rt_attr_float = gpa
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-attr-timestamp"></a>11.2.50.&nbsp;rt_attr_timestamp</h3></div></div></div>
<p>
Timestamp attribute declaration.
Multi-value (an arbitrary number of attributes is allowed), optional.
Introduced in version 1.10-beta.
</p><h4>Example:</h4><pre class="programlisting">
rt_attr_timestamp = date_added
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-rt-attr-string"></a>11.2.51.&nbsp;rt_attr_string</h3></div></div></div>
<p>
String attribute declaration.
Multi-value (an arbitrary number of attributes is allowed), optional.
Introduced in version 1.10-beta.
</p><h4>Example:</h4><pre class="programlisting">
rt_attr_string = author
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="confgroup-indexer"></a>11.3.&nbsp;<code class="filename">indexer</code> program configuration options</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mem-limit"></a>11.3.1.&nbsp;mem_limit</h3></div></div></div>
<p>
Indexing RAM usage limit.
Optional, default is 32M.
</p><p>
Enforced memory usage limit that the <code class="filename">indexer</code>
will not go above. Can be specified in bytes, or kilobytes
(using K postfix), or megabytes (using M postfix); see the example.
This limit will be automatically raised if set to extremely low value
causing I/O buffers to be less than 8 KB; the exact lower bound
for that depends on the indexed data size. If the buffers are
less than 256 KB, a warning will be produced.
</p><p>
Maximum possible limit is 2047M. Too low values can hurt
indexing speed, but 256M to 1024M should be enough for most
if not all datasets. Setting this value too high can cause
SQL server timeouts. During the document collection phase,
there will be periods when the memory buffer is partially
sorted and no communication with the database is performed;
and the database server can timeout. You can resolve that
either by raising timeouts on SQL server side or by lowering
<code class="code">mem_limit</code>.
</p><h4>Example:</h4><pre class="programlisting">
mem_limit = 256M
# mem_limit = 262144K # same, but in KB
# mem_limit = 268435456 # same, but in bytes
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-iops"></a>11.3.2.&nbsp;max_iops</h3></div></div></div>
<p>
Maximum I/O operations per second, for I/O throttling.
Optional, default is 0 (unlimited).
</p><p>
I/O throttling related option.
It limits maximum count of I/O operations (reads or writes) per any given second.
A value of 0 means that no limit is imposed.
</p><p>
<code class="filename">indexer</code> can cause bursts of intensive disk I/O during
indexing, and it might desired to limit its disk activity  (and keep something
for other programs running on the same machine, such as <code class="filename">searchd</code>).
I/O throttling helps to do that. It works by enforcing a minimum guaranteed
delay between subsequent disk I/O operations performed by <code class="filename">indexer</code>. 
Modern SATA HDDs are able to perform up to 70-100+ I/O operations per second
(that's mostly limited by disk heads seek time). Limiting indexing I/O
to a fraction of that can help reduce search performance dedgradation
caused by indexing.
</p><h4>Example:</h4><pre class="programlisting">
max_iops = 40
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-iosize"></a>11.3.3.&nbsp;max_iosize</h3></div></div></div>
<p>
Maximum allowed I/O operation size, in bytes, for I/O throttling.
Optional, default is 0 (unlimited).
</p><p>
I/O throttling related option. It limits maximum file I/O operation
(read or write) size for all operations performed by <code class="filename">indexer</code>.
A value of 0 means that no limit is imposed.
Reads or writes that are bigger than the limit
will be split in several smaller operations, and counted as several operation
by <a href="#conf-max-iops" title="11.3.2.&nbsp;max_iops">max_iops</a> setting. At the time of this
writing, all I/O calls should be under 256 KB (default internal buffer size)
anyway, so <code class="code">max_iosize</code> values higher than 256 KB must not affect anything.
</p><h4>Example:</h4><pre class="programlisting">
max_iosize = 1048576
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-xmlpipe2-field"></a>11.3.4.&nbsp;max_xmlpipe2_field</h3></div></div></div>
<p>
Maximum allowed field size for XMLpipe2 source type, bytes.
Optional, default is 2 MB.
</p><h4>Example:</h4><pre class="programlisting">
max_xmlpipe2_field = 8M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-write-buffer"></a>11.3.5.&nbsp;write_buffer</h3></div></div></div>
<p>
Write buffer size, bytes.
Optional, default is 1 MB.
</p><p>
Write buffers are used to write both temporary and final index
files when indexing. Larger buffers reduce the number of required
disk writes. Memory for the buffers is allocated in addition to
<a href="#conf-mem-limit" title="11.3.1.&nbsp;mem_limit">mem_limit</a>. Note that several
(currently up to 4) buffers for different files will be allocated,
proportionally increasing the RAM usage.
</p><h4>Example:</h4><pre class="programlisting">
write_buffer = 4M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-file-field-buffer"></a>11.3.6.&nbsp;max_file_field_buffer</h3></div></div></div>
<p>
Maximum file field adaptive buffer size, bytes.
Optional, default is 8 MB, minimum is 1 MB.
</p><p>
File field buffer is used to load files referred to from
<a href="#conf-sql-file-field" title="11.1.28.&nbsp;sql_file_field">sql_file_field</a> columns.
This buffer is adaptive, starting at 1 MB at first allocation,
and growing in 2x steps until either file contents can be loaded,
or maximum buffer size, specified by <code class="option">max_file_field_buffer</code>
directive, is reached.
</p><p>
Thus, if there are no file fields are specified, no buffer
is allocated at all.  If all files loaded during indexing are under
(for example) 2 MB in size, but <code class="option">max_file_field_buffer</code> 
value is 128 MB, peak buffer usage would still be only 2 MB. However,
files over 128 MB would be entirely skipped.
</p><h4>Example:</h4><pre class="programlisting">
max_file_field_buffer = 128M
</pre></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="confgroup-searchd"></a>11.4.&nbsp;<code class="filename">searchd</code> program configuration options</h2></div></div></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-listen"></a>11.4.1.&nbsp;listen</h3></div></div></div>
<p>
This setting lets you specify IP address and port, or Unix-domain
socket path, that <code class="code">searchd</code> will listen on.
Introduced in version 0.9.9-rc1.
</p><p>
The informal grammar for <code class="code">listen</code> setting is:
</p><pre class="programlisting">
listen = ( address ":" port | port | path ) [ ":" protocol ]
</pre><p>
I.e. you can specify either an IP address (or hostname) and port
number, or just a port number, or Unix socket path. If you specify
port number but not the address, <code class="code">searchd</code> will listen on
all network interfaces. Unix path is identified by a leading slash.
</p><p>
Starting with version 0.9.9-rc2, you can also specify a protocol
handler (listener) to be used for connections on this socket.
Supported protocol values are 'sphinx' (Sphinx 0.9.x API protocol)
and 'mysql41' (MySQL protocol used since 4.1 upto at least 5.1).
More details on MySQL protocol support can be found in
<a href="#sphinxql" title="5.9.&nbsp;MySQL protocol support and SphinxQL">Section&nbsp;5.9, &#8220;MySQL protocol support and SphinxQL&#8221;</a> section.
</p><h4>Examples:</h4><pre class="programlisting">
listen = localhost
listen = localhost:5000
listen = 192.168.0.1:5000
listen = /var/run/sphinx.s
listen = 9312
listen = localhost:9306:mysql41
</pre><p>
There can be multiple listen directives, <code class="code">searchd</code> will
listen for client connections on all specified ports and sockets.  If
no <code class="code">listen</code> directives are found then the server will listen
on all available interfaces using the default SphinxAPI port 9312.
Starting with 1.10-beta, it will also listen on default SphinxQL 
port 9306. Both port numbers are assigned by IANA (see
<a href="http://www.iana.org/assignments/port-numbers" target="_top">http://www.iana.org/assignments/port-numbers</a>
for details) and should therefore be available.
</p><p>
Unix-domain sockets are not supported on Windows.
</p></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-address"></a>11.4.2.&nbsp;address</h3></div></div></div>
<p>
Interface IP address to bind on.
Optional, default is 0.0.0.0 (ie. listen on all interfaces).
<span class="bold"><strong>DEPRECATED</strong></span>, use <a href="#conf-listen" title="11.4.1.&nbsp;listen">listen</a> instead.
</p><p>
<code class="code">address</code> setting lets you specify which network interface
<code class="filename">searchd</code> will bind to, listen on, and accept incoming
network connections on. The default value is 0.0.0.0 which means to listen
on all interfaces. At the time, you can <span class="bold"><strong>not</strong></span> specify multiple interfaces.
</p><h4>Example:</h4><pre class="programlisting">
address = 192.168.0.1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-port"></a>11.4.3.&nbsp;port</h3></div></div></div>
<p>
<code class="filename">searchd</code> TCP port number.
<span class="bold"><strong>DEPRECATED</strong></span>, use <a href="#conf-listen" title="11.4.1.&nbsp;listen">listen</a> instead.
Used to be mandatory. Default port number is 9312.
</p><h4>Example:</h4><pre class="programlisting">
port = 9312
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-log"></a>11.4.4.&nbsp;log</h3></div></div></div>
<p>
Log file name.
Optional, default is 'searchd.log'.
All <code class="filename">searchd</code> run time events will be logged in this file.
</p><h4>Example:</h4><pre class="programlisting">
log = /var/log/searchd.log
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-query-log"></a>11.4.5.&nbsp;query_log</h3></div></div></div>
<p>
Query log file name.
Optional, default is empty (do not log queries).
All search queries will be logged in this file. The format is described in <a href="#query-log-format" title="5.8.&nbsp;searchd query log format">Section&nbsp;5.8, &#8220;<code class="filename">searchd</code> query log format&#8221;</a>.
</p><h4>Example:</h4><pre class="programlisting">
query_log = /var/log/query.log
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-read-timeout"></a>11.4.6.&nbsp;read_timeout</h3></div></div></div>
<p>
Network client request read timeout, in seconds.
Optional, default is 5 seconds.
<code class="filename">searchd</code> will forcibly close the client connections which fail to send a query within this timeout.
</p><h4>Example:</h4><pre class="programlisting">
read_timeout = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-client-timeout"></a>11.4.7.&nbsp;client_timeout</h3></div></div></div>
<p>
Maximum time to wait between requests (in seconds) when using
persistent connections. Optional, default is five minutes.
</p><h4>Example:</h4><pre class="programlisting">
client_timeout = 3600
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-children"></a>11.4.8.&nbsp;max_children</h3></div></div></div>
<p>
Maximum amount of children to fork (or in other words, concurrent searches to run in parallel).
Optional, default is 0 (unlimited).
</p><p>
Useful to control server load. There will be no more than this much concurrent
searches running, at all times. When the limit is reached, additional incoming
clients are dismissed with temporarily failure (SEARCHD_RETRY) status code
and a message stating that the server is maxed out.
</p><h4>Example:</h4><pre class="programlisting">
max_children = 10
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-pid-file"></a>11.4.9.&nbsp;pid_file</h3></div></div></div>
<p>
<code class="filename">searchd</code> process ID file name.
Mandatory.
</p><p>
PID file will be re-created (and locked) on startup. It will contain
head daemon process ID while the daemon is running, and it will be unlinked
on daemon shutdown. It's mandatory because Sphinx uses it internally
for a number of things: to check whether there already is a running instance
of <code class="filename">searchd</code>; to stop <code class="filename">searchd</code>;
to notify it that it should rotate the indexes. Can also be used for
different external automation scripts.
</p><h4>Example:</h4><pre class="programlisting">
pid_file = /var/run/searchd.pid
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-matches"></a>11.4.10.&nbsp;max_matches</h3></div></div></div>
<p>
Maximum amount of matches that the daemon keeps in RAM for each index and can return to the client.
Optional, default is 1000.
</p><p>
Introduced in order to control and limit RAM usage, <code class="code">max_matches</code>
setting defines how much matches will be kept in RAM while searching each index.
Every match found will still be <span class="emphasis"><em>processed</em></span>; but only
best N of them will be kept in memory and return to the client in the end.
Assume that the index contains 2,000,000 matches for the query. You rarely
(if ever) need to retrieve <span class="emphasis"><em>all</em></span> of them. Rather, you need
to scan all of them, but only choose "best" at most, say, 500 by some criteria
(ie. sorted by relevance, or price, or anything else), and display those
500 matches to the end user in pages of 20 to 100 matches. And tracking
only the best 500 matches is much more RAM and CPU efficient than keeping
all 2,000,000 matches, sorting them, and then discarding everything but
the first 20 needed to display the search results page. <code class="code">max_matches</code>
controls N in that "best N" amount.
</p><p>
This parameter noticeably affects per-query RAM and CPU usage.
Values of 1,000 to 10,000 are generally fine, but higher limits must be
used with care. Recklessly raising <code class="code">max_matches</code> to 1,000,000
means that <code class="filename">searchd</code> will have to allocate and
initialize 1-million-entry matches buffer for <span class="emphasis"><em>every</em></span>
query. That will obviously increase per-query RAM usage, and in some cases
can also noticeably impact performance.
</p><p>
<span class="bold"><strong>CAVEAT EMPTOR!</strong></span> Note that there also is <span class="bold"><strong>another</strong></span> place where this limit
is enforced. <code class="code">max_matches</code> can be decreased on the fly
through the <a href="#api-func-setlimits" title="8.2.1.&nbsp;SetLimits">corresponding API call</a>,
and the default value in the API is <span class="bold"><strong>also</strong></span> set to 1,000. So in order
to retrieve more than 1,000 matches to your application, you will have
to change the configuration file, restart searchd, and set proper limit
in <a href="#api-func-setlimits" title="8.2.1.&nbsp;SetLimits">SetLimits()</a> call.
Also note that you can not set the value in the API higher than the value
in the .conf file. This is prohibited in order to have some protection
against malicious and/or malformed requests.
</p><h4>Example:</h4><pre class="programlisting">
max_matches = 10000
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-seamless-rotate"></a>11.4.11.&nbsp;seamless_rotate</h3></div></div></div>
<p>
Prevents <code class="filename">searchd</code> stalls while rotating indexes with huge amounts of data to precache.
Optional, default is 1 (enable seamless rotation).
</p><p>
Indexes may contain some data that needs to be precached in RAM.
At the moment, <code class="filename">.spa</code>, <code class="filename">.spi</code> and
<code class="filename">.spm</code> files are fully precached (they contain attribute data,
MVA data, and keyword index, respectively.)
Without seamless rotate, rotating an index tries to use as little RAM
as possible and works as follows:
</p><div class="orderedlist"><ol type="1"><li>new queries are temporarly rejected (with "retry" error code);</li>
<li><code class="filename">searchd</code> waits for all currently running queries to finish;</li>
<li>old index is deallocated and its files are renamed;</li>
<li>new index files are renamed and required RAM is allocated;</li>
<li>new index attribute and dictionary data is preloaded to RAM;</li>
<li><code class="filename">searchd</code> resumes serving queries from new index.</li>
</ol></div>
<p>
</p><p>
However, if there's a lot of attribute or dictionary data, then preloading step
could take noticeble time - up to several minutes in case of preloading 1-5+ GB files.
</p><p>
With seamless rotate enabled, rotation works as follows:
</p><div class="orderedlist"><ol type="1"><li>new index RAM storage is allocated;</li>
<li>new index attribute and dictionary data is asynchronously preloaded to RAM;</li>
<li>on success, old index is deallocated and both indexes' files are renamed;</li>
<li>on failure, new index is deallocated;</li>
<li>at any given moment, queries are served either from old or new index copy.</li>
</ol></div>
<p>
</p><p>
Seamless rotate comes at the cost of higher <span class="bold"><strong>peak</strong></span>
memory usage during the rotation (because both old and new copies of
<code class="filename">.spa/.spi/.spm</code> data need to be in RAM while
preloading new copy). Average usage stays the same.
</p><h4>Example:</h4><pre class="programlisting">
seamless_rotate = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-preopen-indexes"></a>11.4.12.&nbsp;preopen_indexes</h3></div></div></div>
<p>
Whether to forcibly preopen all indexes on startup.
Optional, default is 0 (do not preopen).
Enforces enabled <a href="#conf-preopen" title="11.2.32.&nbsp;preopen">preopen</a> on all served indexes,
to avoid manually specifying it in every index.
</p><h4>Example:</h4><pre class="programlisting">
preopen_indexes = 1
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-unlink-old"></a>11.4.13.&nbsp;unlink_old</h3></div></div></div>
<p>
Whether to unlink .old index copies on succesful rotation.
Optional, default is 1 (do unlink).
</p><h4>Example:</h4><pre class="programlisting">
unlink_old = 0
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-attr-flush-period"></a>11.4.14.&nbsp;attr_flush_period</h3></div></div></div>
<p>
When calling <code class="code">UpdateAttributes()</code> to update document attributes in
real-time, changes are first written to the in-memory copy of attributes
(<code class="option">docinfo</code> must be set to <code class="option">extern</code>).
Then, once <code class="filename">searchd</code> shuts down normally (via <code class="code">SIGTERM</code>
being sent), the changes are written to disk.
Introduced in version 0.9.9-rc1.
</p><p>Starting with 0.9.9-rc1, it is possible to tell <code class="filename">searchd</code>
to periodically write these changes back to disk, to avoid them being lost. The time
between those intervals is set with <code class="option">attr_flush_period</code>, in seconds.
</p><p>It defaults to 0, which disables the periodic flushing, but flushing will
still occur at normal shut-down.
</p><h4>Example:</h4><pre class="programlisting">
attr_flush_period = 900 # persist updates to disk every 15 minutes
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-ondisk-dict-default"></a>11.4.15.&nbsp;ondisk_dict_default</h3></div></div></div>
<p>
Instance-wide defaults for <a href="#conf-ondisk-dict" title="11.2.33.&nbsp;ondisk_dict">ondisk_dict</a> directive.
Optional, default it 0 (precache dictionaries in RAM).
Introduced in version 0.9.9-rc1.
</p><p>
This directive lets you specify the default value of
<a href="#conf-ondisk-dict" title="11.2.33.&nbsp;ondisk_dict">ondisk_dict</a> for all the indexes
served by this copy of <code class="filename">searchd</code>. Per-index directive
take precedence, and will overwrite this instance-wide default value,
allowing for fine-grain control.
</p><h4>Example:</h4><pre class="programlisting">
ondisk_dict_default = 1 # keep all dictionaries on disk
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-packet-size"></a>11.4.16.&nbsp;max_packet_size</h3></div></div></div>
<p>
Maximum allowed network packet size.
Limits both query packets from clients, and response packets from remote agents in distributed environment.
Only used for internal sanity checks, does not directly affect RAM use or performance.
Optional, default is 8M.
Introduced in version 0.9.9-rc1.
</p><h4>Example:</h4><pre class="programlisting">
max_packet_size = 32M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-mva-updates-pool"></a>11.4.17.&nbsp;mva_updates_pool</h3></div></div></div>
<p>
Shared pool size for in-memory MVA updates storage.
Optional, default size is 1M.
Introduced in version 0.9.9-rc1.
</p><p>
This setting controls the size of the shared storage pool for updated MVA values.
Specifying 0 for the size disable MVA updates at all. Once the pool size limit
is hit, MVA update attempts will result in an error. However, updates on regular
(scalar) attributes will still work. Due to internal technical difficulties,
currently it is <span class="bold"><strong>not</strong></span> possible to store (flush) <span class="bold"><strong>any</strong></span> updates on indexes
where MVA were updated; though this might be implemented in the future.
In the meantime, MVA updates are intended to be used as a measure to quickly
catchup with latest changes in the database until the next index rebuild;
not as a persistent storage mechanism.
</p><h4>Example:</h4><pre class="programlisting">
mva_updates_pool = 16M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-crash-log-path"></a>11.4.18.&nbsp;crash_log_path</h3></div></div></div>
<p>
Path (formally prefix) for the crash log files.
Optional, default is empty (do not create crash logs).
Introduced in version 0.9.9-rc1.
</p><p>
This is a debugging setting, to help catch rare offending queries
causing crashes without otherwise affecting production instances.
When enabled, <code class="filename">searchd</code> will intercept crash signals
such as SIGSEGV, and dump offending query packets to files named
"crash_log_path.PID", where PID is crashed process ID.
</p><h4>Example:</h4><pre class="programlisting">
crash_log_path = /home/sphinx/log/crashlog
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-filters"></a>11.4.19.&nbsp;max_filters</h3></div></div></div>
<p>
Maximum allowed per-query filter count.
Only used for internal sanity checks, does not directly affect RAM use or performance.
Optional, default is 256.
Introduced in version 0.9.9-rc1.
</p><h4>Example:</h4><pre class="programlisting">
max_filters = 1024
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-filter-values"></a>11.4.20.&nbsp;max_filter_values</h3></div></div></div>
<p>
Maximum allowed per-filter values count.
Only used for internal sanity checks, does not directly affect RAM use or performance.
Optional, default is 4096.
Introduced in version 0.9.9-rc1.
</p><h4>Example:</h4><pre class="programlisting">
max_filter_values = 16384
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-listen-backlog"></a>11.4.21.&nbsp;listen_backlog</h3></div></div></div>
<p>
TCP listen backlog.
Optional, default is 5.
</p><p>
Windows builds currently (as of 0.9.9) can only process the requests
one by one. Concurrent requests will be enqueued by the TCP stack
on OS level, and requests that can not be enqueued will immediately
fail with "connection refused" message. listen_backlog directive
controls the length of the connection queue. Non-Windows builds
should work fine with the default value.
</p><h4>Example:</h4><pre class="programlisting">
listen_backlog = 20
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-read-buffer"></a>11.4.22.&nbsp;read_buffer</h3></div></div></div>
<p>
Per-keyword read buffer size.
Optional, default is 256K.
</p><p>
For every keyword occurrence in every search query, there are
two associated read buffers (one for document list and one for
hit list). This setting lets you control their sizes, increasing
per-query RAM use, but possibly decreasing IO time.
</p><h4>Example:</h4><pre class="programlisting">
read_buffer = 1M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-read-unhinted"></a>11.4.23.&nbsp;read_unhinted</h3></div></div></div>
<p>
Unhinted read size.
Optional, default is 32K.
</p><p>
When querying, some reads know in advance exactly how much data
is there to be read, but some currently do not. Most prominently,
hit list size in not currently known in advance. This setting
lest you control how much data to read in such cases. It will
impact hit list IO time, reducing it for lists larger than
unhinted read size, but raising it for smaller lists. It will
<span class="bold"><strong>not</strong></span> affect RAM use because read buffer will be already
allocated. So it should be not greater than read_buffer.
</p><h4>Example:</h4><pre class="programlisting">
read_unhinted = 32K
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-max-batch-queries"></a>11.4.24.&nbsp;max_batch_queries</h3></div></div></div>
<p>
Limits the amount of queries per batch.
Optional, default is 32.
</p><p>
Makes searchd perform a sanity check of the amount of the queries
submitted in a single batch when using <a href="#multi-queries" title="5.10.&nbsp;Multi-queries">multi-queries</a>.
Set it to 0 to skip the check.
</p><h4>Example:</h4><pre class="programlisting">
max_batch_queries = 256
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-subtree-docs-cache"></a>11.4.25.&nbsp;subtree_docs_cache</h3></div></div></div>
<p>
Max common subtree document cache size, per-query.
Optional, default is 0 (disabled).
</p><p>
Limits RAM usage of a common subtree optimizer (see <a href="#multi-queries" title="5.10.&nbsp;Multi-queries">Section&nbsp;5.10, &#8220;Multi-queries&#8221;</a>).
At most this much RAM will be spent to cache document entries per each query.
Setting the limit to 0 disables the optimizer.
</p><h4>Example:</h4><pre class="programlisting">
subtree_docs_cache = 8M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-subtree-hits-cache"></a>11.4.26.&nbsp;subtree_hits_cache</h3></div></div></div>
<p>
Max common subtree hit cache size, per-query.
Optional, default is 0 (disabled).
</p><p>
Limits RAM usage of a common subtree optimizer (see <a href="#multi-queries" title="5.10.&nbsp;Multi-queries">Section&nbsp;5.10, &#8220;Multi-queries&#8221;</a>).
At most this much RAM will be spent to cache keyword occurrences (hits) per each query.
Setting the limit to 0 disables the optimizer.
</p><h4>Example:</h4><pre class="programlisting">
subtree_hits_cache = 16M
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-workers"></a>11.4.27.&nbsp;workers</h3></div></div></div>
<p>
Multi-processing mode (MPM).
Optional; allowed values are none, fork, prefork, and threads; default is fork.
Introduced in version 1.10-beta.
</p><p>
Lets you choose how <code class="filename">searchd</code> processes multiple
concurrent requests. The possible values are:
</p><div class="variablelist"><dl><dt><span class="term">none</span></dt>
<dd>All requests will be handled serially, one-by-one.
		Prior to 1.x, this was the only mode available on Windows.
	</dd><dt><span class="term">fork</span></dt>
<dd>A new child process will be forked to handle every
		incoming request. Historically, this is the default mode.
	</dd><dt><span class="term">prefork</span></dt>
<dd>On startup, <code class="filename">searchd</code> will pre-fork
		a number of worker processes, and pass the incoming requests
		to one of those children.
	</dd><dt><span class="term">threads</span></dt>
<dd>A new thread will be created to handle every
		incoming request. This is the only mode compatible with
		RT indexing backend.
	</dd></dl></div>
<p>
</p><p>
Historically, <code class="filename">searchd</code> used fork-based model,
which generally performs OK but spends a noticeable amount of CPU
in fork() system call when there's a high amount of (tiny) requests
per second. Prefork mode was implemented to alleviate that; with
prefork, worker processes are basically only created on startup
and re-created on index rotation, somewhat reducing fork() call
pressure.
</p><p>
Threads mode was implemented along with RT backend and is required
to use RT indexes. (Regular disk-based indexes work in all the
available modes.)
</p><h4>Example:</h4><pre class="programlisting">
workers = threads
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-dist-threads"></a>11.4.28.&nbsp;dist_threads</h3></div></div></div>
<p>
Max local search threads to use when searching a distributed index.
Optional, default is 0, which means to disable multi-threaded distributed searching.
Introduced in version 1.10-beta.
</p><p>
Distributed index can include several local indexes. <code class="option">dist_threads</code>
lets you easily utilize multiple CPUs/cores for that (previously existing
alternative was to specify the indexes as remote agents, pointing searchd
to itself and paying some network overheads).
</p><p>
When set to a value N greater than 1, this directive will create up to
N threads for every query, and schedule the specific searches within these
threads. For example, if there are 7 local indexes to search and dist_threads
is set to 2, then 2 parallel threads would be created: one that sequentially
searches 4 indexes, and another one that searches the other 3 indexes.
</p><p>
In case of CPU bound workload, setting <code class="option">dist_threads</code>
to 1x the number of cores is advised (creating more threads than cores
will not improve query time). In case of mixed CPU/disk bound workload
it might sometimes make sense to use more (so that all cores could be
utilizes even when there are threads that wait for I/O completion).
</p><p>
Note that <code class="option">dist_threads</code> does <span class="bold"><strong>not</strong></span> require
threads MPM. You can perfectly use it with fork or prefork MPMs too.
</p><h4>Example:</h4><pre class="programlisting">
index dist_test
{
	type = distributed
	local = chunk1
	local = chunk2
	local = chunk3
	local = chunk4
}

# ...

dist_threads = 4
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-binlog-path"></a>11.4.29.&nbsp;binlog_path</h3></div></div></div>
<p>
Binary log (aka transaction log) files path and name prefix.
Optional, default is empty string, which means to disable logging.
Introduced in version 1.10-beta.
</p><p>
Binary logs are used for crash recovery of RT index data that
would otherwise only be stored in RAM.  When logging is enabled,
every transaction COMMIT-ted into RT index gets written into
a log file.  Logs are then automatically replayed on startup
after an unclean shutdown, recovering the logged changes.
</p><p>
<code class="option">binlog_path</code> directive specifies the binary log
files location.  It should contain both the path and a file name
but not an extension nor a trailing dot; <code class="option">searchd</code>
will append a current log number as an extension (.001, .002, .003,
and so on) to generate a file name.
</p><h4>Example:</h4><pre class="programlisting">
binlog_path = /var/data/binlog # /var/data/binlog.001 etc will be created
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-binlog-flush"></a>11.4.30.&nbsp;binlog_flush</h3></div></div></div>
<p>
Binary log transaction flush/sync mode.
Optional, default is 2 (flush every transaction, sync every second).
Introduced in version 1.10-beta.
</p><p>
This directive controls how frequently will binary log be flushed
to OS and synced to disk. Three modes are supported:
</p><div class="itemizedlist"><ul type="disc"><li>0, flush and sync every second. Best performance,
but up to 1 second worth of committed transactions can be lost
both on daemon crash, or OS/hardware crash.
</li>
<li>1, flush and sync every transaction. Worst performance,
but every committed transaction data is guaranteed to be saved.
</li>
<li>2, flush every transaction, sync every second.
Good performance, and every committed transaction is guaranteed
to be saved in case of daemon crash. However, in case of OS/hardware
crash up to 1 second worth of committed transactions can be lost.
</li>
</ul></div>
<p>
</p><p>
For those familiar with MySQL and InnoDB, this directive is entirely
similar to <code class="option">innodb_flush_log_at_trx_commit</code>. In most
cases, the default hybrid mode 2 provides a nice balance of speed
and safety, with full RT index data protection against daemon crashes,
and some protection against hardware ones.
</p><h4>Example:</h4><pre class="programlisting">
binlog_flush = 1 # ultimate safety, low speed
</pre></div>
<div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="conf-binlog-max-log-size"></a>11.4.31.&nbsp;binlog_max_log_size</h3></div></div></div>
<p>
Maximum binary log file size.
Optional, default is 0 (do not reopen binlog file based on size).
Introduced in version 1.10-beta.
</p><p>
A new binlog file will be forcibly opened once the current binlog file
reaches this limit. This achieves a finer granularity of logs and can yield
more efficient binlog disk usage under certain borderline workloads.
</p><h4>Example:</h4><pre class="programlisting">
binlog_max_log_size = 16M
</pre></div></div></div>
<div class="appendix" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="changelog"></a>Appendix&nbsp;A.&nbsp;Sphinx revision history</h2></div></div></div>
<div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="#rel110">A.1. Version 1.10-beta, 19 jul 2010</a></span></dt>
<dt><span class="sect1"><a href="#rel099">A.2. Version 0.9.9-release, 02 dec 2009</a></span></dt>
<dt><span class="sect1"><a href="#rel099rc2">A.3. Version 0.9.9-rc2, 08 apr 2009</a></span></dt>
<dt><span class="sect1"><a href="#rel099rc1">A.4. Version 0.9.9-rc1, 17 nov 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel0981">A.5. Version 0.9.8.1, 30 oct 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel098">A.6. Version 0.9.8, 14 jul 2008</a></span></dt>
<dt><span class="sect1"><a href="#rel097">A.7. Version 0.9.7, 02 apr 2007</a></span></dt>
<dt><span class="sect1"><a href="#rel097rc2">A.8. Version 0.9.7-rc2, 15 dec 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel097rc">A.9. Version 0.9.7-rc1, 26 oct 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel096">A.10. Version 0.9.6, 24 jul 2006</a></span></dt>
<dt><span class="sect1"><a href="#rel096rc1">A.11. Version 0.9.6-rc1, 26 jun 2006</a></span></dt>
</dl></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel110"></a>A.1.&nbsp;Version 1.10-beta, 19 jul 2010</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added RT indexes support (<a href="#rt-indexes" title="Chapter&nbsp;4.&nbsp;Real-time indexes">Chapter&nbsp;4, <i>Real-time indexes</i></a>)</li>
<li>added prefork and threads support (<a href="#conf-workers" title="11.4.27.&nbsp;workers">workers</a> directives)</li>
<li>added multi-threaded local searches in distributed indexes (<a href="#conf-dist-threads" title="11.4.28.&nbsp;dist_threads">dist_threads</a> directive)</li>
<li>added common subquery cache (<a href="#conf-subtree-docs-cache" title="11.4.25.&nbsp;subtree_docs_cache">subtree_docs_cache</a>,
	<a href="#conf-subtree-hits-cache" title="11.4.26.&nbsp;subtree_hits_cache">subtree_hits_cache</a> directives)</li>
<li>added string attributes support (<a href="#conf-sql-attr-string" title="11.1.24.&nbsp;sql_attr_string">sql_attr_string</a>,
	<a href="#conf-sql-field-string" title="11.1.26.&nbsp;sql_field_string">sql_field_string</a>,
	<a href="#conf-xmlpipe-attr-string" title="11.1.43.&nbsp;xmlpipe_attr_string">xml_attr_string</a>,
	<a href="#conf-xmlpipe-field-string" title="11.1.35.&nbsp;xmlpipe_field_string">xml_field_string</a> directives)</li>
<li>added indexing-time word counter (<a href="#conf-sql-attr-str2wordcount" title="11.1.25.&nbsp;sql_attr_str2wordcount">sql_attr_str2wordcount</a>,
	<a href="#conf-sql-field-str2wordcount" title="11.1.27.&nbsp;sql_field_str2wordcount">sql_field_str2wordcount</a> directives)</li>
<li>added <a href="#sphinxql-call-snippets" title="7.9.&nbsp;CALL SNIPPETS syntax">CALL SNIPPETS()</a>,
	<a href="#sphinxql-call-keywords" title="7.10.&nbsp;CALL KEYWORDS syntax">CALL KEYWORDS()</a> SphinxQL statements</li>
<li>added <code class="option">field_weights</code>, <code class="option">index_weights</code> options to
	SphinxQL <a href="#sphinxql-select" title="7.1.&nbsp;SELECT syntax">SELECT</a> statement</li>
<li>added insert-only SphinxQL-talking tables to SphinxSE (connection='sphinxql://host[:port]/index')</li>
<li>added <code class="option">select</code> option to SphinxSE queries</li>
<li>added backtrace on crash to <code class="filename">searchd</code></li>
<li>added SQL+FS indexing, aka loading files by names fetched from SQL
	(<a href="#conf-sql-file-field" title="11.1.28.&nbsp;sql_file_field">sql_file_field</a> directive)</li>
<li>added a watchdog in threads mode to <code class="filename">searchd</code></li>
<li>added automatic row phantoms elimination to index merge</li>
<li>added hitless indexing support (hitless_words directive)</li>
<li>added --check, --strip-path, --htmlstrip, --dumphitlist ... --wordid switches to <a href="#ref-indextool" title="6.5.&nbsp;indextool command reference">indextool</a></li>
<li>added --stopwait, --logdebug switches to <a href="#ref-searchd" title="6.2.&nbsp;searchd command reference">searchd</a></li>
<li>added --dump-rows, --verbose switches to <a href="#ref-indexer" title="6.1.&nbsp;indexer command reference">indexer</a></li>
<li>added "blended" characters indexing support (<a href="#conf-blend-chars" title="11.2.44.&nbsp;blend_chars">blend_chars</a> directive)</li>
<li>added joined/payload field indexing (<a href="#conf-sql-joined-field" title="11.1.13.&nbsp;sql_joined_field">sql_joined_field</a> directive)</li>
<li>added <a href="#api-func-flushattributes" title="8.7.6.&nbsp;FlushAttributes">FlushAttributes() API call</a></li>
<li>added query_mode, force_all_words, limit_passages, limit_words, start_passage_id, load_files, html_strip_mode,
	allow_empty options, and %PASSAGE_ID% macro in before_match, after_match options
	to <a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">BuildExcerpts()</a> API call</li>
<li>added @groupby/@count/@distinct columns support to SELECT (but not to expressions)</li>
<li>added query-time keyword expansion support (<a href="#conf-expand-keywords" title="11.2.43.&nbsp;expand_keywords">expand_keywords</a> directive,
	<a href="#api-func-setrankingmode" title="8.3.2.&nbsp;SetRankingMode">SPH_RANK_SPH04</a> ranker)</li>
<li>added query batch size limit option (<a href="#conf-max-batch-queries" title="11.4.24.&nbsp;max_batch_queries">max_batch_queries</a> directive; was hardcoded)</li>
<li>added SINT() function to expressions</li>
<li>improved SphinxQL syntax error reporting</li>
<li>improved expression optimizer (better constant handling)</li>
<li>improved dash handling within keywords (no longer treated as an operator)</li>
<li>improved snippets (better passage selection/trimming, around option now a hard limit)</li>
<li>optimized index format that yields ~20-30% smaller indexes</li>
<li>optimized sorting code (indexing time 1-5% faster on average; 100x faster in worst case)</li>
<li>optimized searchd startup time (moved .spa preindexing to indexer), added a progress bar</li>
<li>optimized queries against indexes with many attributes (eliminated redundant copying)</li>
<li>optimized 1-keyword queries (performace regression introduced in 0.9.9)</li>
<li>optimized SphinxQL protocol overheads, and performance on bigger result sets</li>
<li>optimized unbuffered attributes writes on index merge</li>
<li>changed attribute handling, duplicate names are strictly forbidden now</li>
<li>fixed that SphinxQL sessions could stall shutdown</li>
<li>fixed consts with leading minus in SphinxQL</li>
<li>fixed AND/OR precedence in expressions</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=334" target="_top">#334</a>, AVG() on integers was not computed in floats</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=371" target="_top">#371</a>, attribute flush vs 2+ GB files</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=373" target="_top">#373</a>, segfault on distributed queries vs certain libc versions</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=398" target="_top">#398</a>, stopwords not stopped in prefix/infix indexes</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=404" target="_top">#404</a>, erroneous MVA failures in indextool --check</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=408" target="_top">#408</a>, segfault on certain query batches (regular scan, plus a scan with MVA groupby)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=431" target="_top">#431</a>, occasional shutdown hangs in preforked workers</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=436" target="_top">#436</a>, trunk checkout builds vs Solaris sh</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=440" target="_top">#440</a>, escaping vs parentheses declared as valid in charset_table</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=442" target="_top">#442</a>, occasional non-aligned free in MVA indexing</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=447" target="_top">#447</a>, occasional crashes in MVA indexing</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=449" target="_top">#449</a>, pconn busyloop on aborted clients on certain arches</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=465" target="_top">#465</a>, build issue on Alpha</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=468" target="_top">#468</a>, build issue in libsphinxclient</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=472" target="_top">#472</a>, multiple stopword files failing to load</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=489" target="_top">#489</a>, buffer overflow in query logging</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=493" target="_top">#493</a>, Python API assertion after error returned from Query()</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=500" target="_top">#500</a>, malformed MySQL packet when sending MVAs</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=504" target="_top">#504</a>, SIGPIPE in libsphinxclient</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=506" target="_top">#506</a>, better MySQL protocol commands support in SphinxQL (PING etc)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=509" target="_top">#509</a>, indexing ranged results from stored procedures</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel099"></a>A.2.&nbsp;Version 0.9.9-release, 02 dec 2009</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added Open, Close, Status calls to libsphinxclient (C API)</li>
<li>added automatic persistent connection reopening to PHP, Python APIs</li>
<li>added 64-bit value/range filters, fullscan mode support to SphinxSE</li>
<li>MAJOR CHANGE, our IANA assigned ports are 9312 and 9306 respectively (goodbye, trusty 3312)</li>
<li>MAJOR CHANGE, erroneous filters now fail with an error (were silently ignored before)</li>
<li>optimized unbuffered .spa writes on merge</li>
<li>optimized 1-keyword queries ranking in extended2 mode</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=441" target="_top">#441</a> (IO race in case of highly conccurent load on a preopened)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=434" target="_top">#434</a> (distrubuted indexes were not searchable via MySQL protocol)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=317" target="_top">#317</a> (indexer MVA progress counter)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=398" target="_top">#398</a> (stopwords not removed from search query)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=328" target="_top">#328</a> (broken cutoff)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=250" target="_top">#250</a> (now quoting paths w/spaces when installing Windows service)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=348" target="_top">#348</a> (K-list was not updated on merge)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=357" target="_top">#357</a> (destination index were not K-list-filtered on merge)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=369" target="_top">#369</a> (precaching .spi files over 2 GBs)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=438" target="_top">#438</a> (missing boundary proximity matches)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=371" target="_top">#371</a> (.spa flush in case of files over 2 GBs)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=373" target="_top">#373</a> (crashes on distributed queries via mysql proto)</li>
<li>fixed critical bugs in hit merging code</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=424" target="_top">#424</a> (ordinals could be misplaced during indexing in case of bitfields etc)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=426" target="_top">#426</a> (failing SE build on Solaris; thanks to Ben Beecher)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=423" target="_top">#423</a> (typo in SE caused crash on SHOW STATUS)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=363" target="_top">#363</a> (handling of read_timeout over 2147 seconds)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=376" target="_top">#376</a> (minor error message mismatch)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=413" target="_top">#413</a> (minus in SphinxQL)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=417" target="_top">#417</a> (floats w/o leading digit in SphinxQL)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=403" target="_top">#403</a> (typo in SetFieldWeights name in Java API)</li>
<li>fixed index rotation vs persistent connections</li>
<li>fixed backslash handling in SphinxQL parser</li>
<li>fixed uint unpacking vs. PHP 5.2.9 (possibly other versions)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=325" target="_top">#325</a> (filter settings send from SphinxSE)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=352" target="_top">#352</a> (removed mysql wrapper around close() in SphinxSE)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=389" target="_top">#389</a> (display error messages through SphinxSE status variable)</li>
<li>fixed linking with port-installed iconv on OS X</li>
<li>fixed negative 64-bit unpacking in PHP API</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=349" target="_top">#349</a> (escaping backslash in query emulation mode)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=320" target="_top">#320</a> (disabled multi-query route when select items differ)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=353" target="_top">#353</a> (better quorum counts check)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=341" target="_top">#341</a> (merging of trailing hits; maybe other ranking issues too)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=368" target="_top">#368</a> (partially; @field "" caused crashes; now resets field limit)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=365" target="_top">#365</a> (field mask was leaking on field-limited terms)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=339" target="_top">#339</a> (updated debug query dumper)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=361" target="_top">#361</a> (added SetConnectTimeout() to Java API)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=338" target="_top">#338</a> (added missing fullscan to mode check in Java API)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=323" target="_top">#323</a> (added floats support to SphinxQL)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=340" target="_top">#340</a> (support listen=port:proto syntax too)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=332" target="_top">#332</a> (\r is legal SphinxQL space now)</li>
<li>fixed xmlpipe2 K-lists</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=322" target="_top">#322</a> (safety gaps in mysql protocol row buffer)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=313" target="_top">#313</a> (return keyword stats for empty indexes too)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=344" target="_top">#344</a> (invalid checkpoints after merge)</li>
<li>fixed <a href="http://sphinxsearch.com/bugs/view.php?id=326" target="_top">#326</a> (missing CLOCK_xxx on FreeBSD)</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel099rc2"></a>A.3.&nbsp;Version 0.9.9-rc2, 08 apr 2009</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added IsConnectError(), Open(), Close() calls to Java API (bug <a href="http://sphinxsearch.com/bugs/view.php?id=240" target="_top">#240</a>)</li>
<li>added <a href="#conf-read-buffer" title="11.4.22.&nbsp;read_buffer">read_buffer</a>, <a href="#conf-read-unhinted" title="11.4.23.&nbsp;read_unhinted">read_unhinted</a> directives</li>
<li>added checks for build options returned by mysql_config (builds on Solaris now)</li>
<li>added fixed-RAM index merge (bug <a href="http://sphinxsearch.com/bugs/view.php?id=169" target="_top">#169</a>)</li>
<li>added logging chained queries count in case of (optimized) multi-queries</li>
<li>added <a href="#sort-expr">GEODIST()</a> function</li>
<li>added <a href="#ref-searchd" title="6.2.&nbsp;searchd command reference">--status switch to searchd</a></li>
<li>added MySpell (OpenOffice) affix file support (bug <a href="http://sphinxsearch.com/bugs/view.php?id=281" target="_top">#281</a>)</li>
<li>added <a href="#conf-odbc-dsn" title="11.1.10.&nbsp;odbc_dsn">ODBC support</a> (both Windows and UnixODBC)</li>
<li>added support for @id in IN() (bug <a href="http://sphinxsearch.com/bugs/view.php?id=292" target="_top">#292</a>)</li>
<li>added support for <a href="#api-func-setselect" title="8.2.4.&nbsp;SetSelect">aggregate functions</a> in GROUP BY (namely AVG, MAX, MIN, SUM)</li>
<li>added <a href="#sphinxse-snippets" title="9.4.&nbsp;Building snippets (excerpts) via MySQL">MySQL UDF that builds snippets</a> using searchd</li>
<li>added <a href="#conf-write-buffer" title="11.3.5.&nbsp;write_buffer">write_buffer</a> directive (defaults to 1M)</li>
<li>added <a href="#conf-xmlpipe-fixup-utf8" title="11.1.44.&nbsp;xmlpipe_fixup_utf8">xmlpipe_fixup_utf8</a> directive</li>
<li>added suggestions sample</li>
<li>added microsecond precision int64 timer (bug <a href="http://sphinxsearch.com/bugs/view.php?id=282" target="_top">#282</a>)</li>
<li>added <a href="#conf-listen-backlog" title="11.4.21.&nbsp;listen_backlog">listen_backlog directive</a></li>
<li>added <a href="#conf-max-xmlpipe2-field" title="11.3.4.&nbsp;max_xmlpipe2_field">max_xmlpipe2_field</a> directive</li>
<li>added <a href="#sphinxql" title="5.9.&nbsp;MySQL protocol support and SphinxQL">initial SphinxQL support</a> to mysql41 handler, SELECT .../SHOW WARNINGS/STATUS/META are handled</li>
<li>added support for different network protocols, and mysql41 protocol</li>
<li>added <a href="#api-func-setrankingmode" title="8.3.2.&nbsp;SetRankingMode">fieldmask ranker</a>, updated SphinxSE list of rankers</li>
<li>added <a href="#conf-mysql-ssl" title="11.1.9.&nbsp;mysql_ssl_cert, mysql_ssl_key, mysql_ssl_ca">mysql_ssl_xxx</a> directives</li>
<li>added <a href="#ref-searchd" title="6.2.&nbsp;searchd command reference">--cpustats (requires clock_gettime()) and --status switches</a> to searchd</li>
<li>added performance counters, <a href="#api-func-status" title="8.7.5.&nbsp;Status">Status()</a> API call</li>
<li>added <a href="#conf-overshort-step" title="11.2.40.&nbsp;overshort_step">overshort_step</a> and <a href="#conf-stopword-step" title="11.2.41.&nbsp;stopword_step">stopword_step</a> directives</li>
<li>added <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">strict order operator</a> (aka operator before, eg. "one &lt;&lt; two &lt;&lt; three")</li>
<li>added <a href="#ref-indextool" title="6.5.&nbsp;indextool command reference">indextool</a> utility, moved --dumpheader there, added --debugdocids, --dumphitlist options</li>
<li>added own RNG, reseeded on @random sort query (bug <a href="http://sphinxsearch.com/bugs/view.php?id=183" target="_top">#183</a>)</li>
<li>added <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">field-start and field-end modifiers support</a> (syntax is "^hello world$"; field-end requires reindex)</li>
<li>added MVA attribute support to IN() function</li>
<li>added <a href="#sort-expr">AND, OR, and NOT support</a> to expressions</li>
<li>improved logging of (optimized) multi-queries (now logging chained query count)</li>
<li>improved handshake error handling, fixed protocol version byte order (omg)</li>
<li>updated SphinxSE to protocol 1.22</li>
<li>allowed phrase_boundary_step=-1 (trick to emulate keyword expansion)</li>
<li>removed SPH_MAX_QUERY_WORDS limit</li>
<li>fixed CLI search vs documents missing from DB (bug <a href="http://sphinxsearch.com/bugs/view.php?id=257" target="_top">#257</a>)</li>
<li>fixed libsphinxclient results leak on subsequent sphinx_run_queries call (bug <a href="http://sphinxsearch.com/bugs/view.php?id=256" target="_top">#256</a>)</li>
<li>fixed libsphinxclient handling of zero max_matches and cutoff (bug <a href="http://sphinxsearch.com/bugs/view.php?id=208" target="_top">#208</a>)</li>
<li>fixed Java API over-64K string reads (eg. big snippets) in Java API (bug <a href="http://sphinxsearch.com/bugs/view.php?id=181" target="_top">#181</a>)</li>
<li>fixed Java API 2nd Query() after network error in 1st Query() call (bug <a href="http://sphinxsearch.com/bugs/view.php?id=308" target="_top">#308</a>)</li>
<li>fixed typo-class bugs in SetFilterFloatRange (bug <a href="http://sphinxsearch.com/bugs/view.php?id=259" target="_top">#259</a>), SetSortMode (bug #248)</li>
<li>fixed missing @@relaxed support (bug <a href="http://sphinxsearch.com/bugs/view.php?id=276" target="_top">#276</a>), fixed missing error on @nosuchfield queries, documented @@relaxed</li>
<li>fixed UNIX socket permissions to 0777 (bug <a href="http://sphinxsearch.com/bugs/view.php?id=288" target="_top">#288</a>)</li>
<li>fixed xmlpipe2 crash on schemas with no fields, added better document structure checks</li>
<li>fixed (and optimized) expr parser vs IN() with huge (10K+) args count</li>
<li>fixed double EarlyCalc() in fullscan mode (minor performance impact)</li>
<li>fixed phrase boundary handling in some cases (on buffer end, on trailing whitespace)</li>
<li>fixes in snippets (aka excerpts) generation</li>
<li>fixed inline attrs vs id64 index corruption</li>
<li>fixed head searchd crash on config re-parse failure</li>
<li>fixed handling of numeric keywords with leading zeroes such as "007" (bug <a href="http://sphinxsearch.com/bugs/view.php?id=251" target="_top">#251</a>)</li>
<li>fixed junk in SphinxSE status variables (bug <a href="http://sphinxsearch.com/bugs/view.php?id=304" target="_top">#304</a>)</li>
<li>fixed wordlist checkpoints serialization (bug <a href="http://sphinxsearch.com/bugs/view.php?id=236" target="_top">#236</a>)</li>
<li>fixed unaligned docinfo id access (bug <a href="http://sphinxsearch.com/bugs/view.php?id=230" target="_top">#230</a>)</li>
<li>fixed GetRawBytes() vs oversized blocks (headers with over 32K charset_table should now work, bug <a href="http://sphinxsearch.com/bugs/view.php?id=300" target="_top">#300</a>)</li>
<li>fixed buffer overflow caused by too long dest wordform, updated tests</li>
<li>fixed IF() return type (was always int, is deduced now)</li>
<li>fixed legacy queries vs. special chars vs. multiple indexes</li>
<li>fixed write-write-read socket access pattern vs Nagle vs delays vs FreeBSD (oh wow)</li>
<li>fixed exceptions vs query-parser issue</li>
<li>fixed late calc vs @weight in expressions (bug <a href="http://sphinxsearch.com/bugs/view.php?id=285" target="_top">#285</a>)</li>
<li>fixed early lookup/calc vs filters (bug <a href="http://sphinxsearch.com/bugs/view.php?id=284" target="_top">#284</a>)</li>
<li>fixed emulated MATCH_ANY queries (empty proximity and phrase queries are allowed now)</li>
<li>fixed MATCH_ANY ranker vs fields with no matches</li>
<li>fixed index file size vs inplace_enable (bug <a href="http://sphinxsearch.com/bugs/view.php?id=245" target="_top">#245</a>)</li>
<li>fixed that old logs were not closed on USR1 (bug <a href="http://sphinxsearch.com/bugs/view.php?id=221" target="_top">#221</a>)</li>
<li>fixed handling of '!' alias to NOT operator (bug <a href="http://sphinxsearch.com/bugs/view.php?id=237" target="_top">#237</a>)</li>
<li>fixed error handling vs query steps (step failure was not reported)</li>
<li>fixed querying vs inline attributes</li>
<li>fixed stupid bug in escaping code, fixed EscapeString() and made it static</li>
<li>fixed parser vs @field -keyword, foo|@field bar, "" queries (bug <a href="http://sphinxsearch.com/bugs/view.php?id=310" target="_top">#310</a>)</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel099rc1"></a>A.4.&nbsp;Version 0.9.9-rc1, 17 nov 2008</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added <a href="#conf-min-stemming-len" title="11.2.7.&nbsp;min_stemming_len">min_stemming_len</a> directive</li>
<li>added <a href="#api-func-isconnecterror" title="8.1.7.&nbsp;IsConnectError">IsConnectError()</a> API call (helps distingusih API vs remote errors)</li>
<li>added duplicate log messages filter to searchd</li>
<li>added --nodetach debugging switch to searchd</li>
<li>added blackhole agents support for debugging/testing (<a href="#conf-agent-blackhole" title="11.2.29.&nbsp;agent_blackhole">agent_blackhole</a> directive)</li>
<li>added <a href="#conf-max-filters" title="11.4.19.&nbsp;max_filters">max_filters</a>, <a href="#conf-max-filter-values" title="11.4.20.&nbsp;max_filter_values">max_filter_values</a> directives (were hardcoded before)</li>
<li>added int64 expression evaluation path, automatic inference, and BIGINT() enforcer function</li>
<li>added crash handler for debugging (<a href="#conf-crash-log-path" title="11.4.18.&nbsp;crash_log_path">crash_log_path</a> directive)</li>
<li>added MS SQL (aka SQL Server) source support (Windows only, <a href="#conf-mssql-winauth" title="11.1.45.&nbsp;mssql_winauth">mssql_winauth</a> and <a href="#conf-mssql-unicode" title="11.1.46.&nbsp;mssql_unicode">mssql_unicode</a> directives)</li>
<li>added indexer-side column unpacking feature (<a href="#conf-unpack-zlib" title="11.1.47.&nbsp;unpack_zlib">unpack_zlib</a>, <a href="#conf-unpack-mysqlcompress" title="11.1.48.&nbsp;unpack_mysqlcompress">unpack_mysqlcompress</a> directives)</li>
<li>added nested brackers and NOTs support to <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">query language</a>, rewritten query parser</li>
<li>added persistent connections support (<a href="#api-func-open" title="8.8.1.&nbsp;Open">Open()</a> and <a href="#api-func-close" title="8.8.2.&nbsp;Close">Close()</a> API calls)</li>
<li>added <a href="#conf-index-exact-words" title="11.2.39.&nbsp;index_exact_words">index_exact_words</a> feature, and exact form operator to query language ("hello =world")</li>
<li>added status variables support to SphinxSE (SHOW STATUS LIKE 'sphinx_%')</li>
<li>added <a href="#conf-max-packet-size" title="11.4.16.&nbsp;max_packet_size">max_packet_size</a> directive (was hardcoded at 8M before)</li>
<li>added UNIX socket support, and multi-interface support (<a href="#conf-listen" title="11.4.1.&nbsp;listen">listen</a> directive)</li>
<li>added star-syntax support to <a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">BuildExcerpts()</a> API call</li>
<li>added inplace inversion of .spa and .spp (<a href="#conf-inplace-enable" title="11.2.34.&nbsp;inplace_enable">inplace_enable</a> directive, 1.5-2x less disk space for indexing)</li>
<li>added builtin Czech stemmer (morphology=stem_cz)</li>
<li>added <a href="#sort-expr">IDIV(), NOW(), INTERVAL(), IN() functions</a> to expressions</li>
<li>added index-level early-reject based on filters</li>
<li>added MVA updates feature (<a href="#conf-mva-updates-pool" title="11.4.17.&nbsp;mva_updates_pool">mva_updates_pool</a> directive)</li>
<li>added select-list feature with computed expressions support (see <a href="#api-func-setselect" title="8.2.4.&nbsp;SetSelect">SetSelect()</a> API call, test.php --select switch), protocol 1.22</li>
<li>added integer expressions support (2x faster than float)</li>
<li>added multiforms support (multiple source words in wordforms file)</li>
<li>added <a href="#api-func-setrankingmode" title="8.3.2.&nbsp;SetRankingMode">legacy rankers</a> (MATCH_ALL/MATCH_ANY/etc), removed legacy matching code (everything runs on V2 engine now)</li>
<li>added <a href="#extended-syntax" title="5.3.&nbsp;Extended query syntax">field position limit</a> modifier to field operator (syntax: @title[50] hello world)</li>
<li>added killlist support (<a href="#conf-sql-query-killlist" title="11.1.16.&nbsp;sql_query_killlist">sql_query_killlist</a> directive, --merge-killlists switch)</li>
<li>added on-disk SPI support (<a href="#conf-ondisk-dict" title="11.2.33.&nbsp;ondisk_dict">ondisk_dict</a> directive)</li>
<li>added indexer IO stats</li>
<li>added periodic .spa flush (<a href="#conf-attr-flush-period" title="11.4.14.&nbsp;attr_flush_period">attr_flush_period</a> directive)</li>
<li>added config reload on SIGHUP</li>
<li>added per-query attribute overrides feature (see <a href="#api-func-setoverride" title="8.2.3.&nbsp;SetOverride">SetOverride()</a> API call); protocol 1.21</li>
<li>added signed 64bit attrs support (<a href="#conf-sql-attr-bigint" title="11.1.19.&nbsp;sql_attr_bigint">sql_attr_bigint</a> directive)</li>
<li>improved HTML stripper to also skip PIs (&lt;? ... ?&gt;, such as &lt;?php ... ?&gt;)</li>
<li>improved excerpts speed (upto 50x faster on big documents)</li>
<li>fixed a short window of searchd inaccessibility on startup (started listen()ing too early before)</li>
<li>fixed .spa loading on systems where read() is 2GB capped</li>
<li>fixed infixes vs morphology issues</li>
<li>fixed backslash escaping, added backslash to EscapeString()</li>
<li>fixed handling of over-2GB dictionary files (.spi)</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel0981"></a>A.5.&nbsp;Version 0.9.8.1, 30 oct 2008</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added configure script to libsphinxclient</li>
<li>changed proximity/quorum operator syntax to require whitespace after length</li>
<li>fixed potential head process crash on SIGPIPE during "maxed out" message</li>
<li>fixed handling of incomplete remote replies (caused over-degraded distributed results, in rare cases)</li>
<li>fixed sending of big remote requests (caused distributed requests to fail, in rare cases)</li>
<li>fixed FD_SET() overflow (caused searchd to crash on startup, in rare cases)</li>
<li>fixed MVA vs distributed indexes (caused loss of 1st MVA value in result set)</li>
<li>fixed tokenizing of exceptions terminated by specials (eg. "GPS AT&amp;T" in extended mode)</li>
<li>fixed buffer overrun in stemmer on overlong tokens occasionally emitted by proximity/quorum operator parser (caused crashes on certain proximity/quorum queries)</li>
<li>fixed wordcount ranker (could be dropping hits)</li>
<li>fixed --merge feature (numerous different fixes, caused broken indexes)</li>
<li>fixed --merge-dst-range performance</li>
<li>fixed prefix/infix generation for stopwords</li>
<li>fixed ignore_chars vs specials</li>
<li>fixed misplaced F_SETLKW check (caused certain build types, eg. RPM build on FC8, to fail)</li>
<li>fixed dictionary-defined charsets support in spelldump, added \x-style wordchars support</li>
<li>fixed Java API to properly send long strings (over 64K; eg. long document bodies for excerpts)</li>
<li>fixed Python API to accept offset/limit of 'long' type</li>
<li>fixed default ID range (that filtered out all 64-bit values) in Java and Python APIs</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel098"></a>A.6.&nbsp;Version 0.9.8, 14 jul 2008</h2></div></div></div>
<h3>Indexing</h3><div class="itemizedlist"><ul type="disc"><li>added support for 64-bit document and keyword IDs, --enable-id64 switch to configure</li>
<li>added support for floating point attributes</li>
<li>added support for bitfields in attributes, <a href="#conf-sql-attr-bool" title="11.1.18.&nbsp;sql_attr_bool">sql_attr_bool</a> directive and bit-widths part in <a href="#conf-sql-attr-uint" title="11.1.17.&nbsp;sql_attr_uint">sql_attr_uint</a> directive</li>
<li>added support for multi-valued attributes (MVA)</li>
<li>added metaphone preprocessor</li>
<li>added libstemmer library support, provides stemmers for a number of additional languages</li>
<li>added xmlpipe2 source type, that supports arbitrary fields and attributes</li>
<li>added word form dictionaries, <a href="#conf-wordforms" title="11.2.9.&nbsp;wordforms">wordforms</a> directive (and spelldump utility)</li>
<li>added tokenizing exceptions, <a href="#conf-exceptions" title="11.2.10.&nbsp;exceptions">exceptions</a> directive</li>
<li>added an option to fully remove element contents to HTML stripper, <a href="#conf-html-remove-elements" title="11.2.26.&nbsp;html_remove_elements">html_remove_elements</a> directive</li>
<li>added HTML entities decoder (with full XHTML1 set support) to HTML stripper</li>
<li>added per-index HTML stripping settings, <a href="#conf-html-strip" title="11.2.24.&nbsp;html_strip">html_strip</a>, <a href="#conf-html-index-attrs" title="11.2.25.&nbsp;html_index_attrs">html_index_attrs</a>, and <a href="#conf-html-remove-elements" title="11.2.26.&nbsp;html_remove_elements">html_remove_elements</a> directives</li>
<li>added IO load throttling, <a href="#conf-max-iops" title="11.3.2.&nbsp;max_iops">max_iops</a> and <a href="#conf-max-iosize" title="11.3.3.&nbsp;max_iosize">max_iosize</a> directives</li>
<li>added SQL load throttling, <a href="#conf-sql-ranged-throttle" title="11.1.31.&nbsp;sql_ranged_throttle">sql_ranged_throttle</a> directive</li>
<li>added an option to index prefixes/infixes for given fields only, <a href="#conf-prefix-fields" title="11.2.17.&nbsp;prefix_fields">prefix_fields</a> and <a href="#conf-infix-fields" title="11.2.18.&nbsp;infix_fields">infix_fields</a> directives</li>
<li>added an option to ignore certain characters (instead of just treating them as whitespace), <a href="#conf-ignore-chars" title="11.2.14.&nbsp;ignore_chars">ignore_chars</a> directive</li>
<li>added an option to increment word position on phrase boundary characters, <a href="#conf-phrase-boundary" title="11.2.22.&nbsp;phrase_boundary">phrase_boundary</a> and <a href="#conf-phrase-boundary-step" title="11.2.23.&nbsp;phrase_boundary_step">phrase_boundary_step</a> directives</li>
<li>added --merge-dst-range switch (and filters) to index merging feature (--merge switch)</li>
<li>added <a href="#conf-mysql-connect-flags" title="11.1.8.&nbsp;mysql_connect_flags">mysql_connect_flags</a> directive (eg. to reduce indexing time MySQL network traffic and/or time)</li>
<li>improved ordinals sorting; now runs in fixed RAM</li>
<li>improved handling of documents with zero/NULL ids, now skipping them instead of aborting</li>
</ul></div>
<h3>Search daemon</h3><div class="itemizedlist"><ul type="disc"><li>added an option to unlink old index on succesful rotation, <a href="#conf-unlink-old" title="11.4.13.&nbsp;unlink_old">unlink_old</a> directive</li>
<li>added an option to keep index files open at all times (fixes subtle races on rotation), <a href="#conf-preopen" title="11.2.32.&nbsp;preopen">preopen</a> and <a href="#conf-preopen-indexes" title="11.4.12.&nbsp;preopen_indexes">preopen_indexes</a> directives</li>
<li>added an option to profile searchd disk I/O, --iostats command-line option</li>
<li>added an option to rotate index seamlessly (fully avoids query stalls), <a href="#conf-seamless-rotate" title="11.4.11.&nbsp;seamless_rotate">seamless_rotate</a> directive</li>
<li>added HTML stripping support to excerpts (uses per-index settings)</li>
<li>added 'exact_phrase', 'single_passage', 'use_boundaries', 'weight_order 'options to <a href="#api-func-buildexcerpts" title="8.7.1.&nbsp;BuildExcerpts">BuildExcerpts()</a> API call</li>
<li>added distributed attribute updates propagation</li>
<li>added distributed retries on master node side</li>
<li>added log reopen on SIGUSR1</li>
<li>added --stop switch (sends SIGTERM to running instance)</li>
<li>added Windows service mode, and --servicename switch</li>
<li>added Windows --rotate support</li>
<li>improved log timestamping, now with millisecond precision</li>
</ul></div>
<h3>Querying</h3><div class="itemizedlist"><ul type="disc"><li>added extended engine V2 (faster, cleaner, better; SPH_MATCH_EXTENDED2 mode)</li>
<li>added ranking modes support (V2 engine only; <a href="#api-func-setrankingmode" title="8.3.2.&nbsp;SetRankingMode">SetRankingMode()</a> API call)</li>
<li>added quorum searching support to query language (V2 engine only; example: "any three of all these words"/3)</li>
<li>added query escaping support to query language, and <a href="#api-func-escapestring" title="8.7.4.&nbsp;EscapeString">EscapeString()</a> API call</li>
<li>added multi-field syntax support to query language (example: "@(field1,field2) something"), and @@relaxed field checks option</li>
<li>added optional star-syntax ('word*') support in keywords, <a href="#conf-enable-star" title="11.2.19.&nbsp;enable_star">enable_star</a> directive (for prefix/infix indexes only)</li>
<li>added full-scan support (query must be fully empty; can perform block-reject optimization)</li>
<li>added COUNT(DISTINCT(attr)) calculation support, <a href="#api-func-setgroupdistinct" title="8.5.2.&nbsp;SetGroupDistinct">SetGroupDistinct()</a> API call</li>
<li>added group-by on MVA support, <a href="#api-func-setarrayresult" title="8.1.6.&nbsp;SetArrayResult">SetArrayResult()</a> PHP API call</li>
<li>added per-index weights feature, <a href="#api-func-setindexweights" title="8.3.6.&nbsp;SetIndexWeights">SetIndexWeights()</a> API call</li>
<li>added geodistance support, <a href="#api-func-setgeoanchor" title="8.4.5.&nbsp;SetGeoAnchor">SetGeoAnchor()</a> API call</li>
<li>added result set sorting by arbitrary expressions in run time (eg. "@weight+log(price)*2.5"), SPH_SORT_EXPR mode</li>
<li>added result set sorting by @custom compile-time sorting function (see src/sphinxcustomsort.inl)</li>
<li>added result set sorting by @random value</li>
<li>added result set merging for indexes with different schemas</li>
<li>added query comments support (3rd arg to <a href="#api-func-query" title="8.6.1.&nbsp;Query">Query()</a>/<a href="#api-func-addquery" title="8.6.2.&nbsp;AddQuery">AddQuery()</a> API calls, copied verbatim to query log)</li>
<li>added keyword extraction support, <a href="#api-func-buildkeywords" title="8.7.3.&nbsp;BuildKeywords">BuildKeywords()</a> API call</li>
<li>added binding field weights by name, <a href="#api-func-setfieldweights" title="8.3.5.&nbsp;SetFieldWeights">SetFieldWeights()</a> API call</li>
<li>added optional limit on query time, <a href="#api-func-setmaxquerytime" title="8.2.2.&nbsp;SetMaxQueryTime">SetMaxQueryTime()</a> API call</li>
<li>added optional limit on found matches count (4rd arg to <a href="#api-func-setlimits" title="8.2.1.&nbsp;SetLimits">SetLimits()</a> API call, so-called 'cutoff')</li>
</ul></div>
<h3>APIs and SphinxSE</h3><div class="itemizedlist"><ul type="disc"><li>added pure C API (libsphinxclient)</li>
<li>added Ruby API (thanks to Dmytro Shteflyuk)</li>
<li>added Java API</li>
<li>added SphinxSE support for MVAs (use varchar), floats (use float), 64bit docids (use bigint)</li>
<li>added SphinxSE options "floatrange", "geoanchor", "fieldweights", "indexweights", "maxquerytime", "comment", "host" and "port"; and support for "expr:CLAUSE"</li>
<li>improved SphinxSE max query size (using MySQL condition pushdown), upto 256K now</li>
</ul></div>
<h3>General</h3><div class="itemizedlist"><ul type="disc"><li>added scripting (shebang syntax) support to config files (example: #!/usr/bin/php in the first line)</li>
<li>added unified config handling and validation to all programs</li>
<li>added unified documentation </li>
<li>added .spec file for RPM builds</li>
<li>added automated testing suite</li>
<li>improved index locking, now fcntl()-based instead of buggy file-existence-based</li>
<li>fixed unaligned RAM accesses, now works on SPARC and ARM</li>
</ul></div>
<h3><a name="rel098-fixes-since-rc2"></a>Changes and fixes since 0.9.8-rc2</h3><div class="itemizedlist"><ul type="disc"><li>added pure C API (libsphinxclient)</li>
<li>added Ruby API</li>
<li>added SetConnectTimeout() PHP API call</li>
<li>added allowed type check to UpdateAttributes() handler (bug <a href="http://sphinxsearch.com/bugs/view.php?id=174" target="_top">#174</a>)</li>
<li>added defensive MVA checks on index preload (protection against broken indexes, bug <a href="http://sphinxsearch.com/bugs/view.php?id=168" target="_top">#168</a>)</li>
<li>added sphinx-min.conf sample file</li>
<li>added --without-iconv switch to configure</li>
<li>removed redundant -lz dependency in searchd</li>
<li>removed erroneous "xmlpipe2 deprecated" warning</li>
<li>fixed EINTR handling in piped read (bug <a href="http://sphinxsearch.com/bugs/view.php?id=166" target="_top">#166</a>)</li>
<li>fixup query time before logging and sending to client (bug <a href="http://sphinxsearch.com/bugs/view.php?id=153" target="_top">#153</a>)</li>
<li>fixed attribute updates vs full-scan early-reject index (bug <a href="http://sphinxsearch.com/bugs/view.php?id=149" target="_top">#149</a>)</li>
<li>fixed gcc warnings (bug <a href="http://sphinxsearch.com/bugs/view.php?id=160" target="_top">#160</a>)</li>
<li>fixed mysql connection attempt vs pgsql source type (bug <a href="http://sphinxsearch.com/bugs/view.php?id=165" target="_top">#165</a>)</li>
<li>fixed 32-bit wraparound when preloading over 2 GB files</li>
<li>fixed "out of memory" message vs over 2 GB allocs (bug <a href="http://sphinxsearch.com/bugs/view.php?id=116" target="_top">#116</a>)</li>
<li>fixed unaligned RAM access detection on ARM (where unaligned reads do not crash but produce wrong results)</li>
<li>fixed missing full scan results in some cases</li>
<li>fixed several bugs in --merge, --merge-dst-range</li>
<li>fixed @geodist vs MultiQuery and filters, @expr vs MultiQuery</li>
<li>fixed GetTokenEnd() vs 1-grams (was causing crash in excerpts)</li>
<li>fixed sql_query_range to handle empty strings in addition to NULL strings (Postgres specific)</li>
<li>fixed morphology=none vs infixes</li>
<li>fixed case sensitive attributes names in UpdateAttributes()</li>
<li>fixed ext2 ranking vs. stopwords (now using atompos from query parser)</li>
<li>fixed EscapeString() call</li>
<li>fixed escaped specials (now handled as whitespace if not in charset)</li>
<li>fixed schema minimizer (now handles type/size mismatches)</li>
<li>fixed word stats in extended2; stemmed form is now returned</li>
<li>fixed spelldump case folding vs dictionary-defined character sets</li>
<li>fixed Postgres BOOLEAN handling </li>
<li>fixed enforced "inline" docinfo on empty indexes (normally ok, but index merge was really confused)</li>
<li>fixed rare count(distinct) out-of-bounds issue (it occasionaly caused too high @distinct values)</li>
<li>fixed hangups on documents with id=DOCID_MAX in some cases</li>
<li>fixed rare crash in tokenizer (prefixed synonym vs. input stream eof)</li>
<li>fixed query parser vs "aaa (bbb ccc)|ddd" queries</li>
<li>fixed BuildExcerpts() request in Java API</li>
<li>fixed Postgres specific memory leak</li>
<li>fixed handling of overshort keywords (less than min_word_len)</li>
<li>fixed HTML stripper (now emits space after indexed attributes)</li>
<li>fixed 32-field case in query parser</li>
<li>fixed rare count(distinct) vs. querying multiple local indexes vs. reusable sorter issue</li>
<li>fixed sorting of negative floats in SPH_SORT_EXTENDED mode</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel097"></a>A.7.&nbsp;Version 0.9.7, 02 apr 2007</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added support for <code class="option">sql_str2ordinal_column</code></li>
<li>added support for upto 5 sort-by attrs (in extended sorting mode)</li>
<li>added support for separate groups sorting clause (in group-by mode)</li>
<li>added support for on-the-fly attribute updates (PRE-ALPHA; will change heavily; use for preliminary testing ONLY)</li>
<li>added support for zero/NULL attributes</li>
<li>added support for 0.9.7 features to SphinxSE</li>
<li>added support for n-grams (alpha, 1-grams only for now)</li>
<li>added support for warnings reported to client</li>
<li>added support for exclude-filters</li>
<li>added support for prefix and infix indexing (see <code class="option">max_prefix_len</code>, <code class="option">max_infix_len</code>)</li>
<li>added <code class="option">@*</code> syntax to reset current field to query language</li>
<li>added removal of duplicate entries in query index order</li>
<li>added PHP API workarounds for PHP signed/unsigned braindamage</li>
<li>added locks to avoid two concurrent indexers working on same index</li>
<li>added check for existing attributes vs. <code class="option">docinfo=none</code> case</li>
<li>improved groupby code a lot (better precision, and upto 25x times faster in extreme cases)</li>
<li>improved error handling and reporting</li>
<li>improved handling of broken indexes (reports error instead of hanging/crashing)</li>
<li>improved <code class="option">mmap()</code> limits for attributes and wordlists (now able to map over 4 GB on x64 and over 2 GB on x32 where possible)</li>
<li>improved <code class="option">malloc()</code> pressure in head daemon (search time should not degrade with time any more)</li>
<li>improved <code class="filename">test.php</code> command line options</li>
<li>improved error reporting (distributed query, broken index etc issues now reported to client)</li>
<li>changed default network packet size to be 8M, added extra checks</li>
<li>fixed division by zero in BM25 on 1-document collections (in extended matching mode)</li>
<li>fixed <code class="filename">.spl</code> files getting unlinked</li>
<li>fixed crash in schema compatibility test</li>
<li>fixed UTF-8 Russian stemmer</li>
<li>fixed requested matches count when querying distributed agents</li>
<li>fixed signed vs. unsigned issues everywhere (ranged queries, CLI search output, and obtaining docid)</li>
<li>fixed potential crashes vs. negative query offsets</li>
<li>fixed 0-match docs vs. extended mode vs. stats</li>
<li>fixed group/timestamp filters being ignored if querying from older clients</li>
<li>fixed docs to mention <code class="option">pgsql</code> source type</li>
<li>fixed issues with explicit '&amp;' in extended matching mode</li>
<li>fixed wrong assertion in SBCS encoder</li>
<li>fixed crashes with no-attribute indexes after rotate</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel097rc2"></a>A.8.&nbsp;Version 0.9.7-rc2, 15 dec 2006</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added support for extended matching mode (query language)</li>
<li>added support for extended sorting mode (sorting clauses)</li>
<li>added support for SBCS excerpts</li>
<li>added <code class="option">mmap()ing</code> for attributes and wordlist (improves search time, speeds up <code class="option">fork()</code> greatly)</li>
<li>fixed attribute name handling to be case insensitive</li>
<li>fixed default compiler options to simplify post-mortem debugging (added <code class="option">-g</code>, removed <code class="option">-fomit-frame-pointer</code>)</li>
<li>fixed rare memory leak</li>
<li>fixed "hello hello" queries in "match phrase" mode</li>
<li>fixed issue with excerpts, texts and overlong queries</li>
<li>fixed logging multiple index name (no longer tokenized)</li>
<li>fixed trailing stopword not flushed from tokenizer</li>
<li>fixed boolean evaluation</li>
<li>fixed pidfile being wrongly <code class="option">unlink()ed</code> on <code class="option">bind()</code> failure</li>
<li>fixed <code class="option">--with-mysql-includes/libs</code> (they conflicted with well-known paths)</li>
<li>fixes for 64-bit platforms</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel097rc"></a>A.9.&nbsp;Version 0.9.7-rc1, 26 oct 2006</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added alpha index merging code</li>
<li>added an option to decrease <code class="option">max_matches</code> per-query</li>
<li>added an option to specify IP address for searchd to listen on</li>
<li>added support for unlimited amount of configured sources and indexes</li>
<li>added support for group-by queries</li>
<li>added support for /2 range modifier in charset_table</li>
<li>added support for arbitrary amount of document attributes</li>
<li>added logging filter count and index name</li>
<li>added <code class="option">--with-debug</code> option to configure to compile in debug mode</li>
<li>added <code class="option">-DNDEBUG</code> when compiling in default mode</li>
<li>improved search time (added doclist size hints, in-memory wordlist cache, and used VLB coding everywhere)</li>
<li>improved (refactored) SQL driver code (adding new drivers should be very easy now)</li>
<li>improved exceprts generation</li>
<li>fixed issue with empty sources and ranged queries</li>
<li>fixed querying purely remote distributed indexes</li>
<li>fixed suffix length check in English stemmer in some cases</li>
<li>fixed UTF-8 decoder for codes over U+20000 (for CJK)</li>
<li>fixed UTF-8 encoder for 3-byte sequences (for CJK)</li>
<li>fixed overshort (less than <code class="option">min_word_len</code>) words prepended to next field</li>
<li>fixed source connection order (indexer does not connect to all sources at once now)</li>
<li>fixed line numbering in config parser</li>
<li>fixed some issues with index rotation</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel096"></a>A.10.&nbsp;Version 0.9.6, 24 jul 2006</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added support for empty indexes</li>
<li>added support for multiple sql_query_pre/post/post_index</li>
<li>fixed timestamp ranges filter in "match any" mode</li>
<li>fixed configure issues with --without-mysql and --with-pgsql options</li>
<li>fixed building on Solaris 9</li>
</ul></div></div>
<div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="rel096rc1"></a>A.11.&nbsp;Version 0.9.6-rc1, 26 jun 2006</h2></div></div></div>
<div class="itemizedlist"><ul type="disc"><li>added boolean queries support (experimental, beta version)</li>
<li>added simple file-based query cache (experimental, beta version)</li>
<li>added storage engine for MySQL 5.0 and 5.1 (experimental, beta version)</li>
<li>added GNU style <code class="filename">configure</code> script</li>
<li>added new searchd protocol (all binary, and should be backwards compatible)</li>
<li>added distributed searching support to searchd</li>
<li>added PostgreSQL driver</li>
<li>added excerpts generation</li>
<li>added <code class="option">min_word_len</code> option to index</li>
<li>added <code class="option">max_matches</code> option to searchd, removed hardcoded MAX_MATCHES limit</li>
<li>added initial documentation, and a working <code class="filename">example.sql</code></li>
<li>added support for multiple sources per index</li>
<li>added soundex support</li>
<li>added group ID ranges support</li>
<li>added <code class="option">--stdin</code> command-line option to search utility</li>
<li>added <code class="option">--noprogress</code> option to indexer</li>
<li>added <code class="option">--index</code> option to search</li>
<li>fixed UTF-8 decoder (3-byte codepoints did not work)</li>
<li>fixed PHP API to handle big result sets faster</li>
<li>fixed config parser to handle empty values properly</li>
<li>fixed redundant <code class="code">time(NULL)</code> calls in time-segments mode</li>
</ul></div></div></div></div>
</body></html>
